00000800:00000200:3.1F:1761227590.163402:0:2217:0:(socklnd.c:239:ksocknal_find_peer_locked()) got peer_ni [000000001689128d] -> 12345-192.168.201.157@tcp (2) 00000800:00000200:3.1:1761227590.176817:0:2207:0:(socklnd.c:450:ksocknal_incr_conn_count()) Add conn type 1, ksnr_connected 2 ksnr_max_conns 1 00000800:00000200:3.1:1761227590.176829:0:2207:0:(socklnd_cb.c:753:ksocknal_queue_tx_locked()) Sending to 12345-192.168.201.157@tcp ip 192.168.201.157:988 00000800:00000200:3.1:1761227590.176846:0:2207:0:(socklnd_cb.c:772:ksocknal_queue_tx_locked()) Packet 0000000061aaa7ea type 2, nob 96 niov 1 nkiov 0 00000800:00000200:3.1:1761227590.186512:0:2207:0:(socklnd.c:450:ksocknal_incr_conn_count()) Add conn type 2, ksnr_connected 6 ksnr_max_conns 1 00000800:00000200:1.1F:1761227590.192793:0:2207:0:(socklnd.c:450:ksocknal_incr_conn_count()) Add conn type 3, ksnr_connected e ksnr_max_conns 1 00000800:00000100:2.1F:1761231234.993288:0:2211:0:(socklnd_cb.c:2370:ksocknal_find_timed_out_conn()) Timeout sending data to 12345-192.168.201.157@tcp (192.168.201.157:988) the network or that node may be down. 00010000:00010000:0.0:1761232315.057540:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1761232319.153457:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1761232323.249521:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1761232327.345402:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1761232331.441525:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1761232335.537501:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1761232339.633483:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1761232339.889661:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895045952:192.168.201.157@tcp:250:kworker/u8:2.0 00000800:00000100:0.0:1761232339.889747:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232339.889779:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232339/real 1761232339] req@00000000a611ee34 x1846780895045952/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232364 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232339.889872:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a611ee34 x1846780895045952/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232364 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232339.889892:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232339.889895:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232339.889902:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895045952:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761232339.889921:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895046016:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761232339.889952:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232339.889964:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232339/real 1761232339] req@00000000328368c8 x1846780895046016/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232364 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232339.890000:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000328368c8 x1846780895046016/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232364 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232339.890023:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232339.890027:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232339.890037:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895046016:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761232339.890083:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895046080:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232339.890106:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232339.890111:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232339/real 1761232339] req@00000000c652d0ad x1846780895046080/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232364 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232339.890117:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c652d0ad x1846780895046080/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232364 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232339.890122:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232339.890123:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232339.890125:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895046080:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232339.890129:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895046144:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232339.890150:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232339.890162:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232339/real 1761232339] req@000000007ba3340f x1846780895046144/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232364 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232339.890174:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007ba3340f x1846780895046144/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232364 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232339.890181:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232339.890184:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232339.890186:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895046144:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232339.890192:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895046208:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761232339.890247:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232339.890254:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232339/real 1761232339] req@00000000d74b2e3c x1846780895046208/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232364 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232339.890262:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d74b2e3c x1846780895046208/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232364 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232339.890267:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232339.890268:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232339.890270:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895046208:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761232339.890274:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895046272:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232339.890285:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232339.890289:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232339/real 1761232339] req@000000001aedf9b1 x1846780895046272/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232364 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232339.890295:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001aedf9b1 x1846780895046272/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232364 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232339.890299:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232339.890300:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232339.890302:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895046272:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232339.890305:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895046336:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232339.890315:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232339.890318:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232339/real 1761232339] req@00000000568f2a96 x1846780895046336/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232364 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232339.890324:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000568f2a96 x1846780895046336/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232364 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232339.890328:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232339.890330:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232339.890331:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895046336:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1761232343.729510:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1761232345.009688:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895046400:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761232345.009824:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895046464:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761232345.009876:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895046528:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232345.009947:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895046592:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232345.010115:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895046656:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761232345.010191:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895046720:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232345.010383:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895046784:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:0.0:1761232347.825457:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1761232348.081594:0:2209:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:0.0:1761232348.081648:0:2209:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1761232348.081664:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232348.081714:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232348.081724:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232348.081732:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232348.081740:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232348.081748:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232348.081755:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00010000:00010000:0.0:1761232351.921583:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761232355.249422:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232355.249436:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232355.249441:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232355.249482:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 4916 00000100:00080000:0.0:1761232355.249487:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232355.249497:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232355.249545:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895047296/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232355.249582:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232355.249587:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232355.249589:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232355.249594:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4916 00000100:00080000:0.0:1761232355.249597:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232355.249601:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232355.249611:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232355.249626:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895047360/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232355.249638:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232355.249642:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232355.249645:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232355.249649:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4916 00000100:00080000:0.0:1761232355.249652:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232355.249660:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232355.249668:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232355.249688:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895047424/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232355.249704:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232355.249708:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232355.249710:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232355.249713:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4916 00000100:00080000:0.0:1761232355.249716:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232355.249720:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232355.249725:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232355.249739:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895047488/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232355.249750:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232355.249754:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232355.249756:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232355.249760:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4916 00000100:00080000:0.0:1761232355.249762:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232355.249771:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232355.249780:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232355.249795:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895047552/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232355.249806:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232355.249817:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232355.249821:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232355.249835:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4916 00000100:00080000:0.0:1761232355.249839:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232355.249845:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232355.249852:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232355.249870:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895047616/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232355.249894:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232355.249902:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232355.249908:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232355.249917:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4916 00000100:00080000:0.0:1761232355.249925:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232355.249931:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232355.249937:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232355.249956:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895047680/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:0.0:1761232356.017530:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1761232360.113493:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1761232364.209421:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1761232368.305527:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761232370.609471:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232370.609488:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232370.609494:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232370.609509:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 4931 00000100:00080000:0.0:1761232370.609515:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232370.609524:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232370.609578:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895048640/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232370.609638:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232370.609644:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232370.609647:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232370.609653:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4931 00000100:00080000:0.0:1761232370.609658:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232370.609666:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232370.609682:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232370.609711:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895048704/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232370.609733:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232370.609739:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232370.609742:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232370.609749:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4931 00000100:00080000:0.0:1761232370.609754:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232370.609761:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232370.609769:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232370.609788:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895048768/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232370.609814:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232370.609821:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232370.609824:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232370.609833:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4931 00000100:00080000:0.0:1761232370.609840:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232370.609855:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232370.609867:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232370.609896:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895048832/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232370.609931:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232370.609939:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232370.609941:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232370.609951:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4931 00000100:00080000:0.0:1761232370.609957:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232370.609970:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232370.609979:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232370.610007:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895048896/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232370.610075:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232370.610093:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232370.610099:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232370.610115:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4931 00000100:00080000:0.0:1761232370.610126:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232370.610141:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232370.610153:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232370.610185:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895048960/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232370.610324:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232370.610333:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232370.610339:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232370.610356:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4931 00000100:00080000:0.0:1761232370.610364:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232370.610379:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232370.610393:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232370.610427:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895049024/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:0.0:1761232372.401506:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761232375.729396:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232375.729415:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232375.729421:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232375.729438:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 4936 00000100:00080000:0.0:1761232375.729446:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232375.729458:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232375.729509:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895049088/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232375.729546:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232375.729554:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232375.729557:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232375.729563:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4936 00000100:00080000:0.0:1761232375.729567:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232375.729575:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232375.729590:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232375.729614:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895049152/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232375.729642:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232375.729647:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232375.729651:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232375.729658:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4936 00000100:00080000:0.0:1761232375.729662:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232375.729667:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232375.729675:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232375.729689:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895049216/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232375.729702:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232375.729707:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232375.729710:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232375.729715:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4936 00000100:00080000:0.0:1761232375.729719:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232375.729724:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232375.729730:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232375.729745:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895049280/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232375.729760:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232375.729766:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232375.729768:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232375.729774:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4936 00000100:00080000:0.0:1761232375.729778:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232375.729785:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232375.729793:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232375.729811:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895049344/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232375.729826:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232375.729832:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232375.729836:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232375.729842:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4936 00000100:00080000:0.0:1761232375.729845:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232375.729851:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232375.729858:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232375.729872:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895049408/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232375.729888:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232375.729894:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232375.729896:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232375.729902:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4936 00000100:00080000:0.0:1761232375.729906:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232375.729912:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232375.729919:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232375.729935:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895049472/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:0.0:1761232376.497505:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:0.0:1761232380.593474:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761232385.969378:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232385.969386:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232385.969390:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232385.969400:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 4946 00000100:00080000:0.0:1761232385.969403:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232385.969411:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232385.969445:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000addc3db8 x1846780895049984/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232385.969474:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232385.969478:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232385.969480:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232385.969484:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4946 00000100:00080000:0.0:1761232385.969486:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232385.969489:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232385.969498:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232385.969509:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000078e7052d x1846780895050048/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232385.969518:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232385.969521:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232385.969523:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232385.969526:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4946 00000100:00080000:0.0:1761232385.969528:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232385.969534:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232385.969542:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232385.969556:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895050112/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232385.969563:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232385.969566:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232385.969567:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232385.969570:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4946 00000100:00080000:0.0:1761232385.969572:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232385.969574:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232385.969578:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232385.969587:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895050176/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232385.969594:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232385.969597:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232385.969598:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232385.969600:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4946 00000100:00080000:0.0:1761232385.969602:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232385.969605:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232385.969609:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232385.969618:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895050240/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232385.969625:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232385.969630:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232385.969633:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232385.969639:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4946 00000100:00080000:0.0:1761232385.969642:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232385.969646:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232385.969650:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232385.969658:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895050304/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232385.969671:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232385.969680:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232385.969682:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232385.969687:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4946 00000100:00080000:0.0:1761232385.969689:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232385.969696:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232385.969703:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232385.969715:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895050368/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232396.209435:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232396.209447:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232396.209452:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232396.209464:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 4957 00000100:00080000:0.0:1761232396.209469:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232396.209477:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232396.209521:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895050880/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232396.209560:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232396.209566:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232396.209570:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232396.209575:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4957 00000100:00080000:0.0:1761232396.209578:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232396.209583:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232396.209596:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232396.209613:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895050944/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232396.209629:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232396.209642:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232396.209644:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232396.209649:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4957 00000100:00080000:0.0:1761232396.209651:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232396.209655:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232396.209661:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232396.209677:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895051008/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232396.209687:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232396.209691:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232396.209692:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232396.209697:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4957 00000100:00080000:0.0:1761232396.209700:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232396.209704:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232396.209709:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232396.209723:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895051072/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232396.209746:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232396.209752:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232396.209755:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232396.209762:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4957 00000100:00080000:0.0:1761232396.209765:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232396.209774:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232396.209784:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232396.209800:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895051136/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232396.209821:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232396.209834:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232396.209838:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232396.209848:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4957 00000100:00080000:0.0:1761232396.209854:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232396.209866:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232396.209878:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232396.209893:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895051200/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232396.209904:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232396.209908:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232396.209910:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232396.209921:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4957 00000100:00080000:0.0:1761232396.209927:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232396.209932:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232396.209940:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232396.209961:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895051264/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232406.449327:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232406.449335:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232406.449338:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232406.449348:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 4967 00000100:00080000:0.0:1761232406.449351:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232406.449357:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232406.449389:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895051776/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232406.449411:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232406.449414:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232406.449415:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232406.449417:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4967 00000100:00080000:0.0:1761232406.449418:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232406.449421:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232406.449427:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232406.449434:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895051840/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232406.449440:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232406.449443:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232406.449443:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232406.449448:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4967 00000100:00080000:0.0:1761232406.449449:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232406.449452:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232406.449455:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232406.449460:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895051904/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232406.449465:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232406.449466:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232406.449467:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232406.449469:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4967 00000100:00080000:0.0:1761232406.449470:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232406.449473:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232406.449475:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232406.449481:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895051968/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232406.449486:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232406.449488:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232406.449489:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232406.449495:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4967 00000100:00080000:0.0:1761232406.449498:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232406.449501:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232406.449504:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232406.449510:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895052032/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232406.449519:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232406.449522:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232406.449523:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232406.449525:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4967 00000100:00080000:0.0:1761232406.449526:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232406.449528:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232406.449530:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232406.449536:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895052096/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232406.449540:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232406.449542:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232406.449543:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232406.449544:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4967 00000100:00080000:0.0:1761232406.449546:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232406.449548:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232406.449550:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232406.449557:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895052160/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232411.569286:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232411.569295:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232411.569298:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232411.569308:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 4972 00000100:00080000:0.0:1761232411.569311:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232411.569317:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232411.569349:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895052224/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232411.569371:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232411.569373:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232411.569375:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232411.569377:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4972 00000100:00080000:0.0:1761232411.569378:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232411.569381:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232411.569388:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232411.569397:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895052288/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232411.569407:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232411.569410:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232411.569411:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232411.569414:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4972 00000100:00080000:0.0:1761232411.569415:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232411.569417:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232411.569420:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232411.569428:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895052352/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232411.569434:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232411.569436:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232411.569437:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232411.569440:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4972 00000100:00080000:0.0:1761232411.569441:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232411.569444:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232411.569447:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232411.569452:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895052416/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232411.569457:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232411.569458:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232411.569459:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232411.569462:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4972 00000100:00080000:0.0:1761232411.569463:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232411.569467:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232411.569469:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232411.569474:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895052480/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232411.569480:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232411.569482:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232411.569482:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232411.569484:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4972 00000100:00080000:0.0:1761232411.569486:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232411.569491:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232411.569497:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232411.569506:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895052544/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232411.569516:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232411.569521:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232411.569524:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232411.569527:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4972 00000100:00080000:0.0:1761232411.569529:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232411.569532:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232411.569535:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232411.569544:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895052608/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000400:00000100:0.0:1761232414.641554:0:2210:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:0.0:1761232414.641596:0:2210:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1761232414.641609:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232414.641644:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232414.641659:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232414.641667:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232414.641674:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232414.641682:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232414.641689:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00080000:0.0:1761232421.809336:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232421.809352:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232421.809357:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232421.809373:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 4982 00000100:00080000:0.0:1761232421.809377:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232421.809385:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232421.809433:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895053120/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232421.809464:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232421.809469:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232421.809471:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232421.809475:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4982 00000100:00080000:0.0:1761232421.809477:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232421.809481:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232421.809494:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232421.809511:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895053184/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232421.809528:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232421.809540:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232421.809542:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232421.809548:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4982 00000100:00080000:0.0:1761232421.809550:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232421.809556:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232421.809562:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232421.809574:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895053248/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232421.809583:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232421.809588:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232421.809590:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232421.809593:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 4982 00000100:00080000:0.0:1761232421.809596:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232421.809601:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232421.809606:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232421.809620:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895053312/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232421.809634:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232421.809639:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232421.809640:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232421.809648:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4982 00000100:00080000:0.0:1761232421.809657:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232421.809669:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232421.809679:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232421.809709:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895053376/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232421.809720:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232421.809723:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232421.809725:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232421.809740:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4982 00000100:00080000:0.0:1761232421.809745:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232421.809752:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232421.809763:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232421.809782:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895053440/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232421.809795:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232421.809802:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232421.809805:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232421.809812:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 4982 00000100:00080000:0.0:1761232421.809815:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232421.809820:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232421.809826:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232421.809841:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895053504/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232452.529452:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232452.529466:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232452.529471:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232452.529485:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5013 00000100:00080000:0.0:1761232452.529489:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232452.529498:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232452.529545:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895055808/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232452.529582:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232452.529587:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232452.529589:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232452.529594:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5013 00000100:00080000:0.0:1761232452.529596:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232452.529600:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232452.529612:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232452.529627:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895055872/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232452.529641:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232452.529648:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232452.529653:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232452.529664:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5013 00000100:00080000:0.0:1761232452.529667:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232452.529670:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232452.529676:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232452.529693:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895055936/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232452.529703:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232452.529708:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232452.529710:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232452.529715:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5013 00000100:00080000:0.0:1761232452.529718:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232452.529721:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232452.529726:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232452.529740:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895056000/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232452.529758:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232452.529763:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232452.529765:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232452.529773:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5013 00000100:00080000:0.0:1761232452.529775:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232452.529781:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232452.529787:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232452.529803:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895056064/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232452.529819:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232452.529828:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232452.529832:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232452.529846:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5013 00000100:00080000:0.0:1761232452.529851:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232452.529856:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232452.529864:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232452.529880:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895056128/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232452.529894:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232452.529898:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232452.529899:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232452.529906:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5013 00000100:00080000:0.0:1761232452.529908:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232452.529916:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232452.529929:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232452.529949:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895056192/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232457.649406:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232457.649421:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232457.649428:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232457.649443:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5018 00000100:00080000:0.0:1761232457.649448:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232457.649458:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232457.649507:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000078e7052d x1846780895056256/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232457.649541:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232457.649547:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232457.649549:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232457.649553:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5018 00000100:00080000:0.0:1761232457.649556:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232457.649560:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232457.649571:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232457.649586:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000addc3db8 x1846780895056320/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232457.649599:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232457.649604:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232457.649606:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232457.649617:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5018 00000100:00080000:0.0:1761232457.649626:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232457.649633:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232457.649639:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232457.649654:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895056384/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232457.649664:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232457.649669:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232457.649672:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232457.649676:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5018 00000100:00080000:0.0:1761232457.649679:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232457.649683:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232457.649688:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232457.649700:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895056448/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232457.649709:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232457.649713:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232457.649714:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232457.649718:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5018 00000100:00080000:0.0:1761232457.649720:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232457.649731:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232457.649740:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232457.649755:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895056512/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232457.649765:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232457.649773:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232457.649778:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232457.649790:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5018 00000100:00080000:0.0:1761232457.649795:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232457.649805:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232457.649814:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232457.649832:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895056576/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232457.649868:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232457.649880:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232457.649884:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232457.649892:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5018 00000100:00080000:0.0:1761232457.649899:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232457.649910:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232457.649919:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232457.649939:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895056640/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232462.769391:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232462.769408:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232462.769414:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232462.769432:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5023 00000100:00080000:0.0:1761232462.769437:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232462.769448:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232462.769508:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895056704/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232462.769548:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232462.769552:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232462.769555:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232462.769559:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5023 00000100:00080000:0.0:1761232462.769562:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232462.769566:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232462.769582:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232462.769598:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895056768/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232462.769614:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232462.769628:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232462.769632:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232462.769639:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5023 00000100:00080000:0.0:1761232462.769642:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232462.769645:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232462.769651:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232462.769668:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895056832/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232462.769678:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232462.769682:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232462.769685:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232462.769690:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5023 00000100:00080000:0.0:1761232462.769692:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232462.769696:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232462.769702:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232462.769712:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895056896/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232462.769722:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232462.769726:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232462.769727:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232462.769733:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5023 00000100:00080000:0.0:1761232462.769737:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232462.769749:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232462.769756:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232462.769771:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895056960/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232462.769784:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232462.769794:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232462.769798:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232462.769811:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5023 00000100:00080000:0.0:1761232462.769816:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232462.769822:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232462.769827:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232462.769842:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895057024/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232462.769855:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232462.769862:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232462.769868:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232462.769879:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5023 00000100:00080000:0.0:1761232462.769885:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232462.769890:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232462.769895:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232462.769906:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895057088/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232498.609397:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232498.609409:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232498.609413:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232498.609423:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5059 00000100:00080000:0.0:1761232498.609427:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232498.609434:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232498.609471:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895059840/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232498.609498:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232498.609501:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232498.609503:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232498.609506:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5059 00000100:00080000:0.0:1761232498.609508:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232498.609510:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232498.609519:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232498.609531:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895059904/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232498.609542:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232498.609550:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232498.609551:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232498.609555:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5059 00000100:00080000:0.0:1761232498.609556:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232498.609559:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232498.609566:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232498.609576:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895059968/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232498.609583:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232498.609586:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232498.609587:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232498.609590:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5059 00000100:00080000:0.0:1761232498.609591:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232498.609594:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232498.609597:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232498.609604:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895060032/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232498.609610:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232498.609613:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232498.609615:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232498.609623:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5059 00000100:00080000:0.0:1761232498.609627:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232498.609630:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232498.609635:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232498.609647:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895060096/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232498.609683:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232498.609687:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232498.609689:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232498.609698:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5059 00000100:00080000:0.0:1761232498.609703:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232498.609709:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232498.609713:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232498.609726:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895060160/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232498.609734:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232498.609736:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232498.609738:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232498.609740:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5059 00000100:00080000:0.0:1761232498.609742:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232498.609746:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232498.609753:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232498.609769:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895060224/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232503.729372:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232503.729390:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232503.729398:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232503.729416:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5064 00000100:00080000:0.0:1761232503.729422:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232503.729433:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232503.729500:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895060288/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232503.729541:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232503.729547:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232503.729549:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232503.729554:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5064 00000100:00080000:0.0:1761232503.729557:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232503.729561:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232503.729574:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232503.729589:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895060352/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232503.729601:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232503.729606:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232503.729608:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232503.729613:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5064 00000100:00080000:0.0:1761232503.729615:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232503.729619:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232503.729625:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232503.729643:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895060416/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232503.729658:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232503.729661:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232503.729662:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232503.729666:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5064 00000100:00080000:0.0:1761232503.729669:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232503.729672:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232503.729678:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232503.729695:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895060480/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232503.729704:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232503.729709:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232503.729712:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232503.729716:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5064 00000100:00080000:0.0:1761232503.729718:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232503.729722:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232503.729726:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232503.729738:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895060544/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232503.729747:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232503.729751:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232503.729752:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232503.729756:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5064 00000100:00080000:0.0:1761232503.729758:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232503.729795:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232503.729805:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232503.729826:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895060608/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232503.729838:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232503.729843:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232503.729848:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232503.729862:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5064 00000100:00080000:0.0:1761232503.729871:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232503.729880:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232503.729888:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232503.729904:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895060672/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232508.850015:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895061056:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232508.850041:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232508.850052:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232508/real 1761232508] req@00000000ffe86d04 x1846780895061056/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232533 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232508.850064:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ffe86d04 x1846780895061056/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232533 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232508.850071:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232508.850074:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232508.850077:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895061056:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232508.850083:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895061120:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232508.850106:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232508.850115:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232508/real 1761232508] req@00000000d66930c2 x1846780895061120/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232533 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232508.850127:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d66930c2 x1846780895061120/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232533 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232508.850135:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232508.850149:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232508.850153:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895061120:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00080000:2.0:1761232513.969341:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232513.969351:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232513.969354:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232513.969363:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5074 00000100:00080000:2.0:1761232513.969366:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232513.969374:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232513.969407:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895061184/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232513.969431:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232513.969433:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232513.969434:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232513.969437:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5074 00000100:00080000:2.0:1761232513.969438:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232513.969440:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232513.969447:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232513.969455:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895061248/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232513.969461:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232513.969463:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232513.969464:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232513.969466:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5074 00000100:00080000:2.0:1761232513.969468:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232513.969469:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232513.969473:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232513.969481:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895061312/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232513.969489:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232513.969491:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232513.969492:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232513.969494:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5074 00000100:00080000:2.0:1761232513.969496:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232513.969498:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232513.969501:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232513.969507:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895061376/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232513.969512:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232513.969514:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232513.969515:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232513.969518:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5074 00000100:00080000:2.0:1761232513.969519:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232513.969521:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232513.969524:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232513.969530:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895061440/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232513.969534:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232513.969536:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232513.969537:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232513.969540:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5074 00000100:00080000:2.0:1761232513.969541:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232513.969543:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232513.969546:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232513.969553:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895061504/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232513.969564:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232513.969566:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232513.969568:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232513.969570:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5074 00000100:00080000:2.0:1761232513.969571:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232513.969573:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232513.969581:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232513.969590:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895061568/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232529.329444:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232529.329458:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232529.329463:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232529.329505:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5090 00000100:00080000:2.0:1761232529.329510:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232529.329519:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232529.329566:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895062528/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232529.329602:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232529.329607:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232529.329609:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232529.329613:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5090 00000100:00080000:2.0:1761232529.329616:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232529.329620:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232529.329634:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232529.329649:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895062592/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232529.329663:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232529.329671:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232529.329674:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232529.329685:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5090 00000100:00080000:2.0:1761232529.329688:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232529.329691:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232529.329697:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232529.329713:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895062656/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232529.329723:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232529.329727:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232529.329729:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232529.329733:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5090 00000100:00080000:2.0:1761232529.329736:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232529.329741:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232529.329745:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232529.329757:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895062720/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232529.329766:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232529.329771:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232529.329774:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232529.329786:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5090 00000100:00080000:2.0:1761232529.329790:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232529.329795:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232529.329802:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232529.329820:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895062784/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232529.329863:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232529.329867:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232529.329869:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232529.329878:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5090 00000100:00080000:2.0:1761232529.329884:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232529.329894:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232529.329903:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232529.329919:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895062848/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232529.329940:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232529.329949:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232529.329951:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232529.329958:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5090 00000100:00080000:2.0:1761232529.329961:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232529.329965:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232529.329969:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232529.329985:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895062912/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232539.569458:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232539.569477:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232539.569483:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232539.569502:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5100 00000100:00080000:0.0:1761232539.569508:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232539.569519:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232539.569577:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895063424/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232539.569615:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232539.569620:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232539.569622:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232539.569627:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5100 00000100:00080000:0.0:1761232539.569630:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232539.569634:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232539.569647:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232539.569663:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895063488/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232539.569677:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232539.569681:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232539.569685:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232539.569695:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5100 00000100:00080000:0.0:1761232539.569705:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232539.569720:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232539.569728:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232539.569748:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895063552/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232539.569759:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232539.569764:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232539.569767:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232539.569772:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5100 00000100:00080000:0.0:1761232539.569774:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232539.569781:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232539.569787:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232539.569797:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895063616/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232539.569808:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232539.569820:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232539.569824:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232539.569832:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5100 00000100:00080000:0.0:1761232539.569835:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232539.569845:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232539.569853:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232539.569872:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895063680/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232539.569923:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232539.569929:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232539.569935:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232539.569959:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5100 00000100:00080000:0.0:1761232539.569972:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232539.569984:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232539.569993:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232539.570013:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895063744/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232539.570038:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232539.570043:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232539.570046:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232539.570053:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5100 00000100:00080000:0.0:1761232539.570056:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232539.570062:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232539.570074:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232539.570131:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895063808/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232544.689631:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895063872:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761232544.689783:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895063936:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232544.689844:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895064000:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232544.689893:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895064064:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232544.689935:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895064128:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232544.689972:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895064192:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232544.690007:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895064256:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232547.761768:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232544/real 1761232547] req@0000000090ac513e x1846780895064192/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232569 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232547.761797:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090ac513e x1846780895064192/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232569 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232547.761808:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232547.761813:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232547.761817:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895064192:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232547.761823:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232544/real 1761232547] req@0000000053e52331 x1846780895064256/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232569 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232547.761833:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000053e52331 x1846780895064256/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232569 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232547.761840:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232547.761844:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232547.761846:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895064256:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232547.761852:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232544/real 1761232547] req@00000000880f16b4 x1846780895063936/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232569 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232547.761862:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000880f16b4 x1846780895063936/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232569 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232547.761868:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232547.761870:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232547.761873:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895063936:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232547.761878:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232544/real 1761232547] req@00000000cb5be845 x1846780895064000/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232569 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232547.761888:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cb5be845 x1846780895064000/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232569 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232547.761895:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232547.761897:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232547.761900:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895064000:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232547.761905:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232544/real 1761232547] req@00000000a95774fb x1846780895064064/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232569 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232547.761914:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a95774fb x1846780895064064/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232569 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232547.761920:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232547.761923:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232547.761926:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895064064:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232547.761931:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232544/real 1761232547] req@000000000c736bb2 x1846780895064128/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232569 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232547.761940:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000000c736bb2 x1846780895064128/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232569 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232547.761947:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232547.761950:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232547.761952:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895064128:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232547.761957:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232544/real 1761232547] req@00000000038209f5 x1846780895063872/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232569 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232547.761967:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000038209f5 x1846780895063872/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232569 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232547.761975:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232547.761977:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232547.761980:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895063872:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761232549.809704:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895064320:192.168.201.157@tcp:250:kworker/u8:3.0 00000800:00000100:2.0:1761232549.809798:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232549.809838:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232549/real 1761232549] req@00000000a611ee34 x1846780895064320/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232574 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232549.809881:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a611ee34 x1846780895064320/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232574 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232549.809917:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232549.809930:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232549.809945:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895064320:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761232549.809996:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895064384:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761232549.810062:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232549.810091:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232549/real 1761232549] req@00000000328368c8 x1846780895064384/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232574 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232549.810141:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000328368c8 x1846780895064384/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232574 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232549.810176:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232549.810187:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232549.810299:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895064384:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232549.810340:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895064448:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232549.810437:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232549.810462:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232549/real 1761232549] req@00000000c652d0ad x1846780895064448/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232574 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232549.810484:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c652d0ad x1846780895064448/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232574 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232549.810498:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232549.810502:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232549.810508:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895064448:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232549.810541:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895064512:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232549.810585:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232549.810601:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232549/real 1761232549] req@000000007ba3340f x1846780895064512/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232574 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232549.810634:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007ba3340f x1846780895064512/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232574 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232549.810647:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232549.810651:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232549.810656:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895064512:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232549.810665:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895064576:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761232549.810700:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232549.810716:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232549/real 1761232549] req@00000000d74b2e3c x1846780895064576/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232574 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232549.810735:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d74b2e3c x1846780895064576/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232574 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232549.810748:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232549.810752:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232549.810758:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895064576:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232549.810769:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895064640:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232549.810806:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232549.810822:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232549/real 1761232549] req@000000001aedf9b1 x1846780895064640/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232574 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232549.810841:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001aedf9b1 x1846780895064640/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232574 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232549.810856:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232549.810860:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232549.810866:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895064640:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232549.810876:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895064704:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232549.810914:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232549.810929:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232549/real 1761232549] req@00000000568f2a96 x1846780895064704/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232574 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232549.810950:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000568f2a96 x1846780895064704/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232574 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232549.810974:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232549.810978:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232549.810985:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895064704:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1761232554.929374:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232554.929388:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232554.929393:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232554.929406:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5115 00000100:00080000:0.0:1761232554.929411:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232554.929420:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232554.929463:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895064768/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232554.929510:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232554.929515:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232554.929517:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232554.929521:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5115 00000100:00080000:0.0:1761232554.929524:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232554.929529:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232554.929540:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232554.929555:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895064832/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232554.929568:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895064768:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761232554.929572:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232554.929582:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232554.929584:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232554.929589:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5115 00000100:00080000:0.0:1761232554.929592:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232554.929596:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232554.929602:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232554.929616:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895064896/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232554.929626:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232554.929630:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232554.929632:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232554.929639:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5115 00000100:00080000:0.0:1761232554.929642:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232554.929645:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232554.929651:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232554.929664:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895064960/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:2.0:1761232554.929676:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761232554.929685:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232554.929690:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232554.929693:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232554.929700:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5115 00000100:00080000:0.0:1761232554.929703:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232554.929708:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232554.929713:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232554.929714:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232554/real 1761232554] req@000000005593c994 x1846780895064768/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232579 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232554.929730:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895065024/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232554.929746:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1761232554.929751:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895064768/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232579 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232554.929754:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232554.929758:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232554.929770:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5115 00000100:00080000:0.0:1761232554.929775:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232554.929781:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:2.0:1761232554.929781:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232554.929792:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000008:00100000:0.0:1761232554.929794:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232554.929803:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895064768:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761232554.929812:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895065088/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232554.929832:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232554.929836:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232554.929840:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1761232554.929846:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895064832:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:0.0:1761232554.929847:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5115 00000100:00080000:0.0:1761232554.929854:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232554.929862:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232554.929868:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232554.929883:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895065152/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:2.0:1761232554.929896:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232554.929913:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232554/real 1761232554] req@00000000c6e06344 x1846780895064832/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232579 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232554.929930:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895064832/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232579 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232554.929940:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232554.929943:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232554.929946:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895064832:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232554.929953:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895064896:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232554.929973:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232554.929980:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232554/real 1761232554] req@000000004862b808 x1846780895064896/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232579 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232554.929991:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895064896/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232579 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232554.930000:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232554.930002:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232554.930006:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895064896:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232554.930011:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895064960:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232554.930029:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232554.930036:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232554/real 1761232554] req@000000006f2efc8b x1846780895064960/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232579 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232554.930048:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895064960/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232579 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232554.930057:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232554.930059:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232554.930061:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895064960:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232554.930067:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895065024:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232554.930084:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232554.930091:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232554/real 1761232554] req@00000000eca9ddc2 x1846780895065024/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232579 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232554.930102:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895065024/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232579 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232554.930110:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232554.930113:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232554.930116:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895065024:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232554.930123:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895065088:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232554.930140:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232554.930147:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232554/real 1761232554] req@0000000034d49f59 x1846780895065088/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232579 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232554.930158:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895065088/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232579 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232554.930171:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232554.930175:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232554.930178:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895065088:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232554.930261:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895065152:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232554.930288:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232554.930297:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232554/real 1761232554] req@000000005beaae31 x1846780895065152/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232579 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232554.930310:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895065152/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232579 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232554.930318:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232554.930321:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232554.930324:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895065152:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00080000:0.0:1761232565.169470:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232565.169488:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232565.169494:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232565.169511:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5126 00000100:00080000:0.0:1761232565.169516:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232565.169527:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232565.169599:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000addc3db8 x1846780895065664/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232565.169642:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232565.169647:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232565.169649:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232565.169654:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5126 00000100:00080000:0.0:1761232565.169657:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232565.169661:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232565.169677:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232565.169691:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000078e7052d x1846780895065728/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232565.169702:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232565.169706:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232565.169707:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232565.169713:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5126 00000100:00080000:0.0:1761232565.169716:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232565.169721:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232565.169727:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232565.169739:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895065792/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232565.169751:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232565.169755:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232565.169759:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232565.169770:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5126 00000100:00080000:0.0:1761232565.169776:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232565.169791:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232565.169798:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232565.169814:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895065856/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232565.169823:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232565.169828:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232565.169831:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232565.169837:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5126 00000100:00080000:0.0:1761232565.169840:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232565.169848:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232565.169853:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232565.169868:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895065920/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232565.169891:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232565.169895:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232565.169898:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232565.169905:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5126 00000100:00080000:0.0:1761232565.169909:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232565.169916:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232565.169928:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232565.169957:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895065984/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232565.170011:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232565.170016:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232565.170020:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232565.170038:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5126 00000100:00080000:0.0:1761232565.170043:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232565.170054:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232565.170061:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232565.170106:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895066048/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232570.289391:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232570.289405:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232570.289411:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232570.289426:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5131 00000100:00080000:0.0:1761232570.289431:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232570.289441:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232570.289498:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895066112/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232570.289537:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232570.289544:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232570.289547:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232570.289552:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5131 00000100:00080000:0.0:1761232570.289556:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232570.289562:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232570.289575:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232570.289591:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895066176/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232570.289607:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232570.289623:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232570.289625:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232570.289631:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5131 00000100:00080000:0.0:1761232570.289634:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232570.289638:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232570.289647:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232570.289661:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895066240/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232570.289671:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232570.289676:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232570.289678:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232570.289682:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5131 00000100:00080000:0.0:1761232570.289684:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232570.289688:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232570.289694:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232570.289707:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895066304/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232570.289728:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232570.289733:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232570.289735:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232570.289742:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5131 00000100:00080000:0.0:1761232570.289745:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232570.289751:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232570.289759:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232570.289776:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895066368/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232570.289789:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232570.289800:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232570.289804:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232570.289817:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5131 00000100:00080000:0.0:1761232570.289821:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232570.289828:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232570.289836:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232570.289852:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895066432/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232570.289864:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232570.289868:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232570.289870:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232570.289873:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5131 00000100:00080000:0.0:1761232570.289875:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232570.289880:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232570.289886:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232570.289903:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895066496/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761232595.889595:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895068352:192.168.201.157@tcp:250:kworker/u8:3.0 00000800:00000100:0.0:1761232595.889691:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232595.889759:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232595/real 1761232595] req@0000000089b8c4f1 x1846780895068352/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232620 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232595.889788:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895068352/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232620 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232595.889805:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232595.889811:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232595.889817:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895068352:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761232595.889830:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895068416:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761232595.889869:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232595.889884:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232595/real 1761232595] req@0000000057e55ffa x1846780895068416/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232620 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232595.889906:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895068416/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232620 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232595.889920:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232595.889924:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232595.889930:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895068416:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761232595.889969:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895068480:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232595.890009:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232595.890026:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232595/real 1761232595] req@00000000829a5595 x1846780895068480/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232620 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232595.890040:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895068480/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232620 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232595.890050:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232595.890052:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232595.890056:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895068480:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761232595.890062:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895068544:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232595.890085:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232595.890093:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232595/real 1761232595] req@00000000bbf6af7b x1846780895068544/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232620 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232595.890106:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895068544/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232620 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232595.890115:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232595.890117:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232595.890120:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895068544:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761232595.890126:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895068608:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761232595.890145:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232595.890152:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232595/real 1761232595] req@000000002625ec0b x1846780895068608/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232620 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232595.890164:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895068608/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232620 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232595.890172:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232595.890175:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232595.890177:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895068608:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761232595.890183:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895068672:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232595.890259:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232595.890270:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232595/real 1761232595] req@00000000c9993cd4 x1846780895068672/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232620 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232595.890283:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895068672/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232620 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232595.890291:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232595.890294:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232595.890297:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895068672:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761232595.890304:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895068736:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232595.890327:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232595.890335:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232595/real 1761232595] req@000000003b40830d x1846780895068736/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232620 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232595.890347:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895068736/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232620 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232595.890356:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232595.890358:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232595.890361:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895068736:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1761232601.009333:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232601.009347:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232601.009352:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232601.009364:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5161 00000100:00080000:0.0:1761232601.009367:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232601.009374:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232601.009419:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895068800/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232601.009443:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232601.009446:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232601.009448:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232601.009450:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5161 00000100:00080000:0.0:1761232601.009453:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232601.009456:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232601.009465:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232601.009474:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895068864/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232601.009481:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232601.009486:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232601.009488:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232601.009493:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5161 00000100:00080000:0.0:1761232601.009495:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232601.009497:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232601.009500:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232601.009508:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895068928/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232601.009513:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232601.009514:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232601.009515:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232601.009517:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5161 00000100:00080000:0.0:1761232601.009518:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232601.009521:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232601.009524:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232601.009530:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895068992/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232601.009535:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232601.009538:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232601.009539:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232601.009541:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5161 00000100:00080000:0.0:1761232601.009543:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232601.009545:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232601.009547:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232601.009552:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895069056/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232601.009557:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232601.009558:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232601.009559:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232601.009562:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5161 00000100:00080000:0.0:1761232601.009563:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232601.009568:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232601.009572:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232601.009579:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895069120/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232601.009584:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232601.009586:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232601.009587:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232601.009589:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5161 00000100:00080000:0.0:1761232601.009591:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232601.009595:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232601.009601:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232601.009610:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895069184/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232611.249459:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232611.249468:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232611.249472:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232611.249482:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5172 00000100:00080000:0.0:1761232611.249485:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232611.249493:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232611.249525:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895069696/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232611.249569:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232611.249572:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232611.249573:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232611.249576:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5172 00000100:00080000:0.0:1761232611.249577:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232611.249580:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232611.249588:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232611.249597:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895069760/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232611.249601:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895069696:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761232611.249612:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232611.249615:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232611.249616:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232611.249619:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5172 00000100:00080000:0.0:1761232611.249620:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232611.249622:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232611.249628:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232611.249637:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895069824/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232611.249644:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232611.249646:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232611.249647:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232611.249650:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5172 00000100:00080000:0.0:1761232611.249651:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232611.249653:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232611.249656:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232611.249663:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895069888/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232611.249669:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232611.249670:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232611.249671:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232611.249674:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5172 00000100:00080000:0.0:1761232611.249675:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232611.249677:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232611.249680:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232611.249687:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895069952/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232611.249692:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232611.249695:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232611.249696:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1761232611.249698:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895069760:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:0.0:1761232611.249702:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5172 00000100:00080000:0.0:1761232611.249706:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232611.249710:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232611.249715:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232611.249730:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895070016/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232611.249733:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895069824:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1761232611.249745:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232611.249748:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232611.249749:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232611.249752:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5172 00000100:00080000:0.0:1761232611.249753:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232611.249756:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232611.249760:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232611.249764:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895069888:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761232611.249771:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895070080/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232611.249814:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895069952:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232611.249853:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895070016:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232611.249883:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895070080:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232614.322013:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232611/real 1761232614] req@000000005d83493f x1846780895070016/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232636 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232614.322078:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005d83493f x1846780895070016/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232636 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232614.322110:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232614.322118:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232614.322128:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895070016:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232614.322142:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232611/real 1761232614] req@00000000f3ed2588 x1846780895070080/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232636 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232614.322167:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f3ed2588 x1846780895070080/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232636 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232614.322191:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232614.322197:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232614.322254:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895070080:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232614.322268:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232611/real 1761232614] req@000000002d1b8ef7 x1846780895069760/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232636 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232614.322295:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895069760/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232636 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232614.322310:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232614.322318:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232614.322326:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895069760:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232614.322338:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232611/real 1761232614] req@000000004a74da19 x1846780895069824/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232636 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232614.322372:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895069824/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232636 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232614.322385:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232614.322391:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232614.322397:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895069824:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232614.322409:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232611/real 1761232614] req@00000000a4a2c44a x1846780895069888/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232636 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232614.322427:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895069888/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232636 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232614.322440:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232614.322444:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232614.322450:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895069888:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232614.322460:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232611/real 1761232614] req@00000000d67bb589 x1846780895069952/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232636 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232614.322481:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895069952/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232636 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232614.322497:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232614.322537:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232614.322546:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895069952:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232614.322560:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232611/real 1761232614] req@000000009d1c7f73 x1846780895069696/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232636 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232614.322599:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895069696/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232636 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232614.322615:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232614.322619:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232614.322625:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895069696:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761232616.369384:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232616.369401:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232616.369408:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232616.369424:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5177 00000100:00080000:0.0:1761232616.369430:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232616.369439:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232616.369516:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895070144/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232616.369553:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232616.369560:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232616.369562:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232616.369566:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5177 00000100:00080000:0.0:1761232616.369569:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232616.369574:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232616.369588:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232616.369609:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895070208/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232616.369609:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895070144:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761232616.369636:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232616.369641:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232616.369643:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232616.369650:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5177 00000100:00080000:0.0:1761232616.369653:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232616.369656:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232616.369661:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232616.369676:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895070272/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232616.369685:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1761232616.369697:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761232616.369708:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232616.369713:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232616.369723:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5177 00000100:00080000:0.0:1761232616.369727:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1761232616.369732:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232616/real 1761232616] req@000000001db96463 x1846780895070144/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232641 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232616.369750:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232616.369762:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232616.369771:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895070144/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232641 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232616.369789:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895070336/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232616.369790:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232616.369799:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232616.369814:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895070144:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761232616.369826:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232616.369835:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232616.369838:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232616.369846:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5177 00000100:00080000:0.0:1761232616.369850:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232616.369859:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:2.0:1761232616.369865:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895070208:192.168.201.157@tcp:38:kworker/u8:3.0 00000008:00100000:0.0:1761232616.369872:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232616.369902:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895070400/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:2.0:1761232616.369913:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232616.369929:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232616/real 1761232616] req@00000000ef655201 x1846780895070208/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232641 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232616.369965:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1761232616.369984:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895070208/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232641 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232616.369986:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232616.369993:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232616.369999:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232616.370009:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5177 00000100:00080000:2.0:1761232616.370013:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761232616.370017:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1761232616.370026:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895070208:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:0.0:1761232616.370033:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232616.370052:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232616.370057:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895070272:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761232616.370085:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895070464/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232616.370106:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1761232616.370107:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761232616.370118:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232616.370123:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1761232616.370131:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232616/real 1761232616] req@000000006d61dd06 x1846780895070272/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232641 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232616.370135:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5177 00000100:00080000:0.0:1761232616.370155:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232616.370171:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:2.0:1761232616.370183:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895070272/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232641 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:0.0:1761232616.370185:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1761232616.370209:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232616.370214:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232616.370223:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895070272:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761232616.370280:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895070528/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232616.370296:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895070336:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232616.370344:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232616.370364:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232616/real 1761232616] req@00000000efa29eb9 x1846780895070336/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232641 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232616.370378:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895070336/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232641 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232616.370387:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232616.370391:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232616.370395:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895070336:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232616.370436:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895070400:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761232616.370466:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232616.370475:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232616/real 1761232616] req@000000001f9ec359 x1846780895070400/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232641 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232616.370510:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895070400/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232641 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232616.370518:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232616.370520:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232616.370523:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895070400:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232616.370529:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895070464:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232616.370551:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232616.370560:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232616/real 1761232616] req@000000004130b6bd x1846780895070464/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232641 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232616.370570:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895070464/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232641 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232616.370578:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232616.370580:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232616.370583:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895070464:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232616.370589:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895070528:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232616.370607:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232616.370614:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232616/real 1761232616] req@000000007bd1ec64 x1846780895070528/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232641 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232616.370624:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895070528/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232641 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232616.370632:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232616.370635:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232616.370637:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895070528:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232621.489536:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac65ecb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895070592:192.168.201.157@tcp:250:kworker/u8:2.0 00000800:00000100:2.0:1761232621.489614:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232621.489640:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232621/real 1761232621] req@00000000ac65ecb7 x1846780895070592/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232646 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232621.489682:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac65ecb7 x1846780895070592/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232646 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232621.489700:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232621.489705:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232621.489712:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac65ecb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895070592:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761232621.489744:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a5e9865 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895070656:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232621.489776:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232621.489786:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232621/real 1761232621] req@000000006a5e9865 x1846780895070656/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232646 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232621.489795:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006a5e9865 x1846780895070656/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232646 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232621.489802:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232621.489803:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232621.489806:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a5e9865 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895070656:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232621.489811:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004361f727 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895070720:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232621.489825:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232621.489830:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232621/real 1761232621] req@000000004361f727 x1846780895070720/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232646 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232621.489839:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004361f727 x1846780895070720/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232646 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232621.489845:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232621.489846:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232621.489848:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004361f727 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895070720:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232621.489852:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895070784:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232621.489864:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232621.489868:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232621/real 1761232621] req@00000000df0b94a8 x1846780895070784/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232646 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232621.489876:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df0b94a8 x1846780895070784/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232646 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232621.489882:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232621.489883:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232621.489885:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895070784:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232621.489889:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895070848:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232621.489902:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232621.489906:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232621/real 1761232621] req@00000000df5b2270 x1846780895070848/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232646 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232621.489915:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df5b2270 x1846780895070848/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232646 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232621.489921:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232621.489922:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232621.489924:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895070848:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232621.489928:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895070912:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232621.489939:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232621.489944:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232621/real 1761232621] req@00000000ffe86d04 x1846780895070912/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232646 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232621.489951:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ffe86d04 x1846780895070912/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232646 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232621.489956:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232621.489958:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232621.489961:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895070912:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232621.489979:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895070976:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232621.489992:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232621.489997:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232621/real 1761232621] req@00000000d66930c2 x1846780895070976/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232646 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232621.490005:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d66930c2 x1846780895070976/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232646 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232621.490011:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232621.490013:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232621.490015:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895070976:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00080000:2.0:1761232626.609292:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232626.609302:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232626.609307:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232626.609319:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5187 00000100:00080000:2.0:1761232626.609323:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232626.609330:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232626.609384:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895071040/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232626.609411:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232626.609414:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232626.609418:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232626.609421:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5187 00000100:00080000:2.0:1761232626.609422:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232626.609425:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232626.609434:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232626.609442:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895071104/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232626.609448:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232626.609450:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232626.609451:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232626.609454:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5187 00000100:00080000:2.0:1761232626.609455:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232626.609458:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232626.609461:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232626.609469:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895071168/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232626.609475:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232626.609477:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232626.609479:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232626.609481:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5187 00000100:00080000:2.0:1761232626.609482:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232626.609485:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232626.609488:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232626.609494:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895071232/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232626.609500:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232626.609507:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232626.609511:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232626.609521:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5187 00000100:00080000:2.0:1761232626.609523:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232626.609525:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232626.609527:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232626.609537:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895071296/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232626.609542:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232626.609544:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232626.609546:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232626.609550:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5187 00000100:00080000:2.0:1761232626.609552:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232626.609555:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232626.609558:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232626.609565:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895071360/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232626.609569:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232626.609571:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232626.609572:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232626.609574:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5187 00000100:00080000:2.0:1761232626.609576:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232626.609578:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232626.609581:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232626.609594:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895071424/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232631.729440:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232631.729465:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232631.729472:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232631.729495:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5192 00000100:00080000:2.0:1761232631.729504:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232631.729519:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232631.729597:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895071488/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232631.729663:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232631.729674:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232631.729678:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232631.729688:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5192 00000100:00080000:2.0:1761232631.729693:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232631.729701:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232631.729721:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232631.729736:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895071488:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761232631.729753:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895071552/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232631.729780:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232631.729788:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232631.729791:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232631.729800:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5192 00000100:00080000:2.0:1761232631.729805:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232631.729813:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232631.729826:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1761232631.729843:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232631.729854:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895071616/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232631.729879:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1761232631.729881:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232631/real 1761232631] req@0000000099b7e5d9 x1846780895071488/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232656 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232631.729900:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232631.729906:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232631.729925:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5192 00000100:00100000:0.0:1761232631.729926:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895071488/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232656 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232631.729947:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232631.729959:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232631.729969:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761232631.729969:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:0.0:1761232631.729982:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895071488:192.168.201.157@tcp:250:kworker/u8:3.0 00000008:00100000:2.0:1761232631.729990:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232631.730033:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895071552:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232631.730034:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895071680/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232631.730072:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1761232631.730080:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761232631.730091:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232631.730099:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1761232631.730101:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232631/real 1761232631] req@00000000f4e415d3 x1846780895071552/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232656 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232631.730122:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5192 00000100:00080000:2.0:1761232631.730131:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232631.730139:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:0.0:1761232631.730153:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895071552/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232656 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:2.0:1761232631.730155:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1761232631.730179:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232631.730183:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232631.730195:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895071744/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761232631.730197:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895071552:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761232631.730241:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895071616:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761232631.730270:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1761232631.730274:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761232631.730283:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232631.730291:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1761232631.730300:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232631/real 1761232631] req@000000008ebccab5 x1846780895071616/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232656 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232631.730302:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5192 00000100:00080000:2.0:1761232631.730330:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232631.730347:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232631.730370:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232631.730379:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895071616/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232656 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232631.730392:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1761232631.730395:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895071808/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232631.730398:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232631.730417:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895071616:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761232631.730439:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232631.730446:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232631.730451:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232631.730460:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5192 00000100:00080000:2.0:1761232631.730464:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1761232631.730473:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895071680:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761232631.730473:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232631.730490:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232631.730508:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895071872/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:0.0:1761232631.730526:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232631.730557:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232631/real 1761232631] req@00000000a1e8c6bb x1846780895071680/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232656 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232631.730584:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895071680/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232656 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232631.730593:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232631.730596:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232631.730600:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895071680:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761232631.730607:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895071744:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761232631.730628:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232631.730636:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232631/real 1761232631] req@00000000c10cebe6 x1846780895071744/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232656 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232631.730648:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895071744/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232656 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232631.730657:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232631.730659:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232631.730662:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895071744:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761232631.730669:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895071808:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232631.730687:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232631.730694:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232631/real 1761232631] req@00000000ac7d8d5b x1846780895071808/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232656 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232631.730706:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895071808/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232656 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232631.730714:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232631.730717:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232631.730720:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895071808:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761232631.730747:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895071872:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232631.730766:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232631.730774:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232631/real 1761232631] req@000000004ae5bc9b x1846780895071872/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232656 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232631.730785:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895071872/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232656 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232631.730793:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232631.730796:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232631.730799:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895071872:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761232636.849429:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232636.849448:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232636.849456:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232636.849474:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5197 00000100:00080000:2.0:1761232636.849481:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232636.849493:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232636.849558:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895071936/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232636.849593:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232636.849598:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232636.849600:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232636.849605:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5197 00000100:00080000:2.0:1761232636.849608:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232636.849612:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232636.849627:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232636.849646:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895072000/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232636.849659:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232636.849663:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232636.849667:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1761232636.849672:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895071936:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761232636.849683:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5197 00000100:00080000:2.0:1761232636.849694:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232636.849703:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232636.849715:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232636.849748:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895072064/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232636.849768:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232636.849777:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1761232636.849784:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761232636.849785:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232636.849800:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5197 00000100:00080000:2.0:1761232636.849806:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232636.849814:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:0.0:1761232636.849821:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232636/real 1761232636] req@0000000088939481 x1846780895071936/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232661 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:2.0:1761232636.849828:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232636.849850:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895071936/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232661 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232636.849866:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1761232636.849867:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895072128/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232636.849872:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232636.849883:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895071936:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761232636.849908:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232636.849916:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232636.849920:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1761232636.849925:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895072000:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:2.0:1761232636.849935:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5197 00000100:00080000:2.0:1761232636.849942:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232636.849950:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:0.0:1761232636.849961:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:2.0:1761232636.849962:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232636.849975:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232636/real 1761232636] req@000000006ff1fdbc x1846780895072000/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232661 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232636.849990:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895072192/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761232636.849997:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895072000/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232661 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232636.850073:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232636.850076:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232636.850079:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761232636.850093:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1761232636.850100:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895072000:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:2.0:1761232636.850101:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232636.850122:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5197 00000100:00100000:0.0:1761232636.850124:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895072064:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761232636.850139:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232636.850159:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:0.0:1761232636.850173:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:2.0:1761232636.850175:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232636.850187:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232636/real 1761232636] req@00000000135e9a6e x1846780895072064/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232661 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232636.850243:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895072064/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232661 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232636.850279:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232636.850282:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232636.850288:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895072064:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232636.850296:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895072256/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761232636.850307:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895072128:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761232636.850324:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232636.850330:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232636.850334:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232636.850341:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5197 00000100:00080000:2.0:1761232636.850346:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1761232636.850355:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761232636.850360:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232636.850369:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232636.850371:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232636/real 1761232636] req@00000000fd3935a6 x1846780895072128/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232661 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232636.850414:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895072128/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232661 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232636.850415:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895072320/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232636.850500:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232636.850508:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232636.850514:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895072128:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761232636.850546:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895072192:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761232636.850568:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232636.850576:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232636/real 1761232636] req@00000000fd4aa3a2 x1846780895072192/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232661 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232636.850587:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895072192/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232661 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232636.850596:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232636.850599:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232636.850602:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895072192:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761232636.850608:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895072256:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232636.850625:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232636.850632:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232636/real 1761232636] req@000000006c1041ba x1846780895072256/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232661 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232636.850643:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895072256/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232661 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232636.850651:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232636.850653:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232636.850655:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895072256:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761232636.850661:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895072320:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232636.850677:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232636.850683:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232636/real 1761232636] req@000000006d8c3eb0 x1846780895072320/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232661 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232636.850693:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895072320/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232661 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232636.850701:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232636.850703:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232636.850706:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895072320:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761232641.969326:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232641.969338:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232641.969342:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232641.969354:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5202 00000100:00080000:2.0:1761232641.969358:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232641.969366:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232641.969411:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895072384/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232641.969436:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232641.969439:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232641.969440:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232641.969442:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5202 00000100:00080000:2.0:1761232641.969444:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232641.969446:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232641.969454:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232641.969463:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895072448/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232641.969470:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1761232641.969473:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895072384:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761232641.969473:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232641.969475:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232641.969479:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5202 00000100:00080000:2.0:1761232641.969480:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232641.969484:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232641.969489:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232641.969496:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895072512/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232641.969500:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232641.969503:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232641.969504:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232641.969507:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5202 00000100:00080000:2.0:1761232641.969509:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232641.969511:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232641.969514:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232641.969524:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895072576/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232641.969528:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232641.969531:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232641.969532:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232641.969535:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5202 00000100:00080000:2.0:1761232641.969536:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232641.969540:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232641.969542:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232641.969548:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895072640/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232641.969553:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232641.969555:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232641.969556:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232641.969558:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5202 00000100:00080000:2.0:1761232641.969559:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232641.969562:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:0.0:1761232641.969566:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:2.0:1761232641.969567:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232641.969576:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895072704/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232641.969585:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232641.969588:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232641.969589:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232641.969592:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5202 00000100:00100000:0.0:1761232641.969595:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232641/real 1761232641] req@000000004f29ca26 x1846780895072384/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232666 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232641.969595:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232641.969599:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232641.969605:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232641.969616:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895072768/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761232641.969621:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895072384/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232666 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232641.969632:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232641.969634:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232641.969639:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895072384:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761232641.969663:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895072448:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761232641.969677:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232641.969682:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232641/real 1761232641] req@0000000033765045 x1846780895072448/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232666 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232641.969689:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895072448/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232666 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232641.969693:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232641.969694:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232641.969696:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895072448:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761232641.969699:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895072512:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232641.969711:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232641.969715:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232641/real 1761232641] req@000000009d315fa5 x1846780895072512/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232666 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232641.969726:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895072512/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232666 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232641.969731:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232641.969732:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232641.969733:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895072512:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761232641.969736:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895072576:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232641.969745:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232641.969749:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232641/real 1761232641] req@0000000040d5f361 x1846780895072576/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232666 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232641.969754:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895072576/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232666 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232641.969758:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232641.969759:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232641.969761:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895072576:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761232641.969764:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895072640:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761232641.969773:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232641.969776:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232641/real 1761232641] req@000000007dcf84ae x1846780895072640/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232666 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232641.969781:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895072640/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232666 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232641.969785:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232641.969787:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232641.969789:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895072640:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761232641.969793:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895072704:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232641.969802:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232641.969806:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232641/real 1761232641] req@000000007091f547 x1846780895072704/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232666 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232641.969811:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895072704/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232666 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232641.969816:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232641.969817:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232641.969819:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895072704:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761232641.969822:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895072768:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232641.969830:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232641.969834:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232641/real 1761232641] req@0000000064a2cea0 x1846780895072768/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232666 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232641.969839:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895072768/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232666 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232641.969843:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232641.969844:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232641.969846:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895072768:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761232647.089303:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232647.089319:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232647.089324:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232647.089339:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5208 00000100:00080000:2.0:1761232647.089345:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232647.089355:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232647.089406:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895072832/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232647.089442:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232647.089446:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232647.089448:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232647.089453:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5208 00000100:00080000:2.0:1761232647.089455:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232647.089459:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232647.089475:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232647.089490:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895072896/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761232647.089501:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895072832:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761232647.089509:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232647.089523:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232647.089525:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232647.089532:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5208 00000100:00080000:2.0:1761232647.089535:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232647.089538:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232647.089544:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232647.089561:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895072960/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232647.089571:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232647.089574:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232647.089576:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232647.089581:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5208 00000100:00080000:2.0:1761232647.089584:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232647.089587:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232647.089592:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232647.089606:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895073024/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:0.0:1761232647.089617:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761232647.089627:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232647.089631:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232647.089634:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232647.089643:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5208 00000100:00080000:2.0:1761232647.089646:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232647.089650:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232647.089655:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232647.089657:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232647/real 1761232647] req@0000000064a2cea0 x1846780895072832/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232672 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232647.089671:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895073088/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232647.089686:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1761232647.089692:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895072832/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232672 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232647.089694:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232647.089698:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232647.089713:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5208 00000100:00080000:2.0:1761232647.089719:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232647.089723:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232647.089724:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:0.0:1761232647.089736:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000008:00100000:2.0:1761232647.089738:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232647.089748:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895072832:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761232647.089756:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895073152/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232647.089769:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232647.089772:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232647.089775:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232647.089779:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5208 00000100:00080000:2.0:1761232647.089782:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232647.089787:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232647.089792:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232647.089793:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895072896:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232647.089807:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895073216/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:0.0:1761232647.089831:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232647.089846:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232647/real 1761232647] req@000000007091f547 x1846780895072896/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232672 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232647.089862:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895072896/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232672 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232647.089871:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232647.089874:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232647.089878:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895072896:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761232647.089884:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895072960:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232647.089904:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232647.089911:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232647/real 1761232647] req@000000007dcf84ae x1846780895072960/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232672 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232647.089923:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895072960/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232672 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232647.089933:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232647.089935:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232647.089939:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895072960:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761232647.089944:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895073024:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232647.089962:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232647.089969:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232647/real 1761232647] req@0000000040d5f361 x1846780895073024/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232672 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232647.089980:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895073024/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232672 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232647.089989:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232647.089991:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232647.089994:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895073024:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761232647.089999:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895073088:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761232647.090017:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232647.090024:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232647/real 1761232647] req@000000009d315fa5 x1846780895073088/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232672 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232647.090035:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895073088/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232672 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232647.090043:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232647.090046:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232647.090049:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895073088:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761232647.090056:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895073152:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232647.090073:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232647.090080:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232647/real 1761232647] req@0000000033765045 x1846780895073152/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232672 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232647.090092:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895073152/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232672 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232647.090100:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232647.090103:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232647.090106:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895073152:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761232647.090155:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895073216:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232647.090179:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232647.090189:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232647/real 1761232647] req@000000004f29ca26 x1846780895073216/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232672 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232647.090233:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895073216/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232672 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232647.090243:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232647.090245:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232647.090249:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895073216:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761232652.209383:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232652.209398:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232652.209404:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232652.209417:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5213 00000100:00080000:2.0:1761232652.209422:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232652.209432:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232652.209476:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895073280/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232652.209508:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232652.209514:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232652.209516:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232652.209521:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5213 00000100:00080000:2.0:1761232652.209525:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232652.209529:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232652.209540:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232652.209555:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895073344/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232652.209567:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232652.209571:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1761232652.209576:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895073280:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761232652.209577:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232652.209590:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5213 00000100:00080000:2.0:1761232652.209594:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232652.209597:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232652.209607:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232652.209632:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895073408/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232652.209643:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232652.209647:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232652.209649:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232652.209652:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5213 00000100:00080000:2.0:1761232652.209655:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232652.209659:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232652.209663:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232652.209677:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895073472/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:0.0:1761232652.209691:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761232652.209692:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232652.209699:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232652.209701:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232652.209707:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5213 00000100:00080000:2.0:1761232652.209711:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232652.209716:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232652.209722:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232652.209728:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232652/real 1761232652] req@000000006d8c3eb0 x1846780895073280/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232677 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232652.209740:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895073536/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232652.209766:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232652.209771:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232652.209775:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1761232652.209776:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895073280/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232677 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232652.209790:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5213 00000100:00080000:2.0:1761232652.209798:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232652.209805:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:0.0:1761232652.209806:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1761232652.209810:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1761232652.209813:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232652.209823:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895073280:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761232652.209831:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895073600/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232652.209843:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232652.209847:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232652.209849:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232652.209853:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5213 00000100:00080000:2.0:1761232652.209855:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232652.209860:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:0.0:1761232652.209866:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895073344:192.168.201.157@tcp:38:kworker/u8:3.0 00000008:00100000:2.0:1761232652.209867:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232652.209884:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895073664/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:0.0:1761232652.209908:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232652.209924:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232652/real 1761232652] req@000000006c1041ba x1846780895073344/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232677 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232652.209938:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895073344/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232677 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232652.209948:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232652.209951:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232652.209954:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895073344:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761232652.209961:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895073408:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232652.209980:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232652.209988:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232652/real 1761232652] req@00000000fd4aa3a2 x1846780895073408/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232677 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232652.210000:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895073408/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232677 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232652.210008:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232652.210011:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232652.210014:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895073408:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761232652.210021:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895073472:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232652.210038:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232652.210045:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232652/real 1761232652] req@00000000fd3935a6 x1846780895073472/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232677 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232652.210056:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895073472/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232677 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232652.210065:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232652.210067:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232652.210070:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895073472:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761232652.210124:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895073536:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761232652.210148:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232652.210157:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232652/real 1761232652] req@00000000135e9a6e x1846780895073536/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232677 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232652.210169:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895073536/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232677 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232652.210178:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232652.210180:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232652.210194:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895073536:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761232652.210230:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895073600:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232652.210259:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232652.210275:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232652/real 1761232652] req@000000006ff1fdbc x1846780895073600/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232677 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232652.210288:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895073600/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232677 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232652.210296:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232652.210299:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232652.210302:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895073600:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761232652.210330:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895073664:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232652.210349:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232652.210357:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232652/real 1761232652] req@0000000088939481 x1846780895073664/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232677 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232652.210369:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895073664/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232677 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232652.210378:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232652.210380:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232652.210383:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895073664:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1761232657.329381:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232657.329394:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232657.329399:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232657.329411:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5218 00000100:00080000:0.0:1761232657.329416:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232657.329425:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232657.329467:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895073728/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232657.329501:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232657.329505:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232657.329506:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232657.329510:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5218 00000100:00080000:0.0:1761232657.329512:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232657.329515:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232657.329525:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232657.329535:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895073792/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232657.329543:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232657.329546:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232657.329547:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232657.329550:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5218 00000100:00080000:0.0:1761232657.329552:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232657.329554:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232657.329558:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232657.329564:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895073856/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232657.329571:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232657.329573:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232657.329574:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232657.329578:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5218 00000100:00080000:0.0:1761232657.329582:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232657.329586:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232657.329591:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232657.329601:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895073920/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232657.329607:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232657.329610:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232657.329612:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232657.329618:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5218 00000100:00080000:0.0:1761232657.329621:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232657.329625:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232657.329629:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232657.329640:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895073984/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232657.329656:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232657.329670:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232657.329671:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232657.329676:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5218 00000100:00080000:0.0:1761232657.329678:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232657.329681:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232657.329685:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232657.329698:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895074048/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232657.329712:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232657.329717:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232657.329719:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232657.329722:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5218 00000100:00080000:0.0:1761232657.329723:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232657.329727:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232657.329730:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232657.329742:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895074112/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761232667.569775:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895074624:192.168.201.157@tcp:250:kworker/u8:2.0 00000800:00000100:0.0:1761232667.569884:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232667.569950:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232667/real 1761232667] req@000000009706a782 x1846780895074624/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232692 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232667.570004:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895074624/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232692 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232667.570018:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232667.570023:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232667.570029:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895074624:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761232667.570039:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895074688:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761232667.570067:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232667.570076:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232667/real 1761232667] req@000000007aee79ba x1846780895074688/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232692 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232667.570099:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895074688/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232692 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232667.570107:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232667.570110:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232667.570113:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895074688:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761232667.570120:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895074752:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232667.570140:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232667.570147:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232667/real 1761232667] req@000000008e58c4b8 x1846780895074752/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232692 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232667.570159:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895074752/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232692 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232667.570168:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232667.570170:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232667.570173:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895074752:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232667.570256:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895074816:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232667.570282:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232667.570292:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232667/real 1761232667] req@0000000080131ea7 x1846780895074816/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232692 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232667.570304:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895074816/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232692 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232667.570312:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232667.570316:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232667.570319:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895074816:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232667.570325:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895074880:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761232667.570344:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232667.570352:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232667/real 1761232667] req@000000007adf99ab x1846780895074880/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232692 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232667.570364:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895074880/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232692 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232667.570372:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232667.570374:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232667.570377:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895074880:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761232667.570383:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895074944:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232667.570399:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232667.570407:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232667/real 1761232667] req@00000000e99e5a12 x1846780895074944/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232692 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232667.570417:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895074944/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232692 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232667.570425:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232667.570427:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232667.570430:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895074944:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232667.570436:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895075008:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232667.570457:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232667.570464:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232667/real 1761232667] req@0000000070fa3bd6 x1846780895075008/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232692 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232667.570475:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895075008/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232692 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232667.570483:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232667.570485:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232667.570488:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895075008:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232672.689431:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895075072:192.168.201.157@tcp:250:kworker/u8:2.0 00000800:00000100:0.0:1761232672.689497:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232672.689522:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232672/real 1761232672] req@00000000a611ee34 x1846780895075072/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232697 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232672.689544:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a611ee34 x1846780895075072/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232697 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232672.689566:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232672.689571:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232672.689578:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895075072:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761232672.689603:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895075136:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761232672.689624:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232672.689631:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232672/real 1761232672] req@00000000328368c8 x1846780895075136/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232697 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232672.689639:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000328368c8 x1846780895075136/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232697 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232672.689645:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232672.689647:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232672.689649:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895075136:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761232672.689654:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895075200:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232672.689666:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232672.689670:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232672/real 1761232672] req@00000000c652d0ad x1846780895075200/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232697 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232672.689676:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c652d0ad x1846780895075200/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232697 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232672.689681:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232672.689682:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232672.689683:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895075200:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232672.689686:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895075264:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232672.689696:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232672.689699:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232672/real 1761232672] req@000000007ba3340f x1846780895075264/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232697 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232672.689705:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007ba3340f x1846780895075264/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232697 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232672.689709:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232672.689711:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232672.689712:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895075264:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232672.689716:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895075328:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761232672.689725:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232672.689729:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232672/real 1761232672] req@00000000d74b2e3c x1846780895075328/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232697 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232672.689735:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d74b2e3c x1846780895075328/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232697 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232672.689739:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232672.689741:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232672.689742:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895075328:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761232672.689745:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895075392:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232672.689754:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232672.689758:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232672/real 1761232672] req@000000001aedf9b1 x1846780895075392/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232697 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232672.689763:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001aedf9b1 x1846780895075392/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232697 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232672.689767:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232672.689768:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232672.689770:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895075392:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232672.689784:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895075456:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232672.689793:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232672.689797:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232672/real 1761232672] req@00000000568f2a96 x1846780895075456/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232697 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232672.689804:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000568f2a96 x1846780895075456/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232697 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232672.689808:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232672.689809:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232672.689811:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895075456:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00080000:2.0:1761232677.809445:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232677.809460:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232677.809467:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232677.809481:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5238 00000100:00080000:2.0:1761232677.809486:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232677.809497:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232677.809546:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895075520/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232677.809584:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232677.809591:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232677.809593:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232677.809597:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5238 00000100:00080000:2.0:1761232677.809600:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232677.809604:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232677.809617:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232677.809631:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895075584/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232677.809644:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232677.809648:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232677.809650:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232677.809658:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5238 00000100:00080000:2.0:1761232677.809662:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1761232677.809667:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895075520:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761232677.809675:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232677.809689:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232677.809703:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895075648/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232677.809717:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232677.809724:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232677.809728:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232677.809733:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5238 00000100:00080000:2.0:1761232677.809735:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232677.809739:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232677.809744:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232677.809757:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895075712/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232677.809772:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232677.809776:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232677.809778:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232677.809811:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5238 00000100:00080000:2.0:1761232677.809813:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232677.809818:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232677.809827:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232677.809846:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895075776/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232677.809858:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232677.809862:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232677.809864:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232677.809869:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5238 00000100:00080000:2.0:1761232677.809873:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232677.809882:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232677.809890:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232677.809906:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895075584:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232677.809915:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895075840/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232677.809941:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232677.809951:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232677.809955:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1761232677.809962:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895075648:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761232677.809969:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5238 00000100:00080000:2.0:1761232677.809979:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232677.809987:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232677.809996:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232677.810004:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895075712:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232677.810025:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895075904/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761232677.810040:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895075776:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761232677.810090:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895075840:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761232677.810128:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895075904:192.168.201.157@tcp:8:kworker/u8:3.0 00000400:00000100:0.0:1761232680.881790:0:2210:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:0.0:1761232680.881853:0:2210:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1761232680.881871:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232680.881931:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232680.881942:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232680.881955:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232680.881968:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232680.881984:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232680.881998:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:0.0:1761232682.929493:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbad49de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895075968:192.168.201.157@tcp:250:kworker/u8:2.0 00000800:00000100:0.0:1761232682.929572:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232682.929599:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232682/real 1761232682] req@00000000cbad49de x1846780895075968/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232707 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232682.929633:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cbad49de x1846780895075968/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232707 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232682.929645:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232682.929648:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232682.929652:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbad49de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895075968:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761232682.929659:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e467e107 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895076032:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761232682.929681:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232682.929689:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232682/real 1761232682] req@00000000e467e107 x1846780895076032/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232707 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232682.929701:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e467e107 x1846780895076032/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232707 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232682.929709:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232682.929711:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232682.929715:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e467e107 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895076032:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761232682.929775:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090800dde pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895076096:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232682.929796:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232682.929804:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232682/real 1761232682] req@0000000090800dde x1846780895076096/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232707 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232682.929815:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090800dde x1846780895076096/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232707 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232682.929823:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232682.929826:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232682.929829:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090800dde pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895076096:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232682.929835:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895076160:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232682.929852:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232682.929859:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232682/real 1761232682] req@00000000568f2a96 x1846780895076160/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232707 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232682.929869:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000568f2a96 x1846780895076160/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232707 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232682.929877:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232682.929879:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232682.929882:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895076160:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232682.929888:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895076224:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761232682.929912:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232682.929919:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232682/real 1761232682] req@000000001aedf9b1 x1846780895076224/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232707 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232682.929930:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001aedf9b1 x1846780895076224/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232707 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232682.929944:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232682.929946:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232682.929949:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895076224:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761232682.929955:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895076288:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232682.929974:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232682.929981:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232682/real 1761232682] req@00000000d74b2e3c x1846780895076288/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232707 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232682.929993:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d74b2e3c x1846780895076288/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232707 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232682.930000:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232682.930002:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232682.930005:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895076288:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232682.930011:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895076352:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232682.930030:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232682.930037:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232682/real 1761232682] req@000000007ba3340f x1846780895076352/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232707 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232682.930048:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007ba3340f x1846780895076352/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232707 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232682.930056:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232682.930058:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232682.930061:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895076352:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232688.049599:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895076416:192.168.201.157@tcp:250:kworker/u8:2.0 00000800:00000100:0.0:1761232688.049739:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232688.049774:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232688/real 1761232688] req@00000000b6fdd7f9 x1846780895076416/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232713 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232688.049819:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895076416/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232713 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232688.049869:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232688.049879:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232688.049894:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895076416:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761232688.049945:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895076480:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761232688.050012:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232688.050031:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232688/real 1761232688] req@0000000090702bc4 x1846780895076480/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232713 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232688.050054:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895076480/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232713 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232688.050063:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232688.050067:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232688.050070:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895076480:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761232688.050078:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895076544:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232688.050105:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232688.050114:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232688/real 1761232688] req@0000000070fa3bd6 x1846780895076544/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232713 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232688.050150:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895076544/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232713 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232688.050157:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232688.050158:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232688.050161:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895076544:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232688.050166:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895076608:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232688.050182:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232688.050188:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232688/real 1761232688] req@00000000e99e5a12 x1846780895076608/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232713 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232688.050198:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895076608/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232713 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232688.050205:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232688.050206:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232688.050208:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895076608:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232688.050213:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895076672:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761232688.050227:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232688.050262:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232688/real 1761232688] req@000000007adf99ab x1846780895076672/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232713 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232688.050271:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895076672/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232713 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232688.050278:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232688.050281:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232688.050284:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895076672:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761232688.050289:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895076736:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232688.050307:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232688.050314:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232688/real 1761232688] req@0000000080131ea7 x1846780895076736/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232713 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232688.050323:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895076736/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232713 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232688.050330:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232688.050332:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232688.050334:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895076736:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232688.050338:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895076800:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232688.050352:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232688.050358:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232688/real 1761232688] req@000000008e58c4b8 x1846780895076800/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232713 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232688.050367:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895076800/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232713 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232688.050373:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232688.050375:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232688.050378:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895076800:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00080000:0.0:1761232693.169344:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232693.169358:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232693.169363:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232693.169374:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5254 00000100:00080000:0.0:1761232693.169377:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232693.169385:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232693.169422:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000078e7052d x1846780895076864/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232693.169446:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232693.169450:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232693.169451:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232693.169455:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5254 00000100:00080000:0.0:1761232693.169456:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232693.169460:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232693.169469:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232693.169481:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000addc3db8 x1846780895076928/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232693.169490:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232693.169495:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232693.169497:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232693.169505:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5254 00000100:00080000:0.0:1761232693.169507:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232693.169510:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232693.169514:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232693.169526:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895076992/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232693.169535:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232693.169538:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232693.169539:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232693.169542:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5254 00000100:00080000:0.0:1761232693.169543:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232693.169546:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232693.169549:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232693.169557:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895077056/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232693.169563:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232693.169565:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232693.169566:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232693.169569:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5254 00000100:00080000:0.0:1761232693.169571:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232693.169579:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232693.169584:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232693.169595:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895077120/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232693.169603:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232693.169633:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232693.169636:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232693.169643:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5254 00000100:00080000:0.0:1761232693.169645:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232693.169650:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232693.169657:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232693.169674:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895077184/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232693.169693:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232693.169697:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232693.169698:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232693.169703:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5254 00000100:00080000:0.0:1761232693.169704:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232693.169707:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232693.169710:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232693.169722:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895077248/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232703.409313:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232703.409323:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232703.409326:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232703.409336:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5264 00000100:00080000:0.0:1761232703.409339:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232703.409345:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232703.409375:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895077760/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232703.409399:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232703.409401:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232703.409402:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232703.409405:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5264 00000100:00080000:0.0:1761232703.409407:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232703.409410:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232703.409417:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232703.409424:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895077824/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232703.409429:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232703.409431:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232703.409432:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232703.409434:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5264 00000100:00080000:0.0:1761232703.409435:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232703.409437:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232703.409440:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232703.409444:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895077888/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232703.409448:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232703.409450:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232703.409451:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232703.409453:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5264 00000100:00080000:0.0:1761232703.409455:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232703.409457:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232703.409459:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232703.409465:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895077952/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232703.409473:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232703.409476:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232703.409477:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232703.409480:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5264 00000100:00080000:0.0:1761232703.409481:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232703.409483:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232703.409486:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232703.409505:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895078016/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232703.409511:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232703.409517:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232703.409518:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232703.409521:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5264 00000100:00080000:0.0:1761232703.409522:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232703.409524:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232703.409527:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232703.409534:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895078080/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232703.409539:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232703.409542:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232703.409543:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232703.409545:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5264 00000100:00080000:0.0:1761232703.409546:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232703.409549:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232703.409552:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232703.409558:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895078144/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232708.529319:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232708.529329:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232708.529332:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232708.529340:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5269 00000100:00080000:2.0:1761232708.529343:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232708.529349:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232708.529381:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895078208/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232708.529403:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232708.529405:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232708.529406:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232708.529409:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5269 00000100:00080000:2.0:1761232708.529410:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232708.529412:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232708.529418:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232708.529425:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895078272/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232708.529430:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232708.529447:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232708.529449:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232708.529452:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5269 00000100:00080000:2.0:1761232708.529453:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232708.529456:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232708.529459:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232708.529465:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895078336/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232708.529470:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232708.529473:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232708.529474:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232708.529476:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5269 00000100:00080000:2.0:1761232708.529478:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232708.529480:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232708.529482:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232708.529487:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895078400/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232708.529493:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232708.529495:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232708.529496:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232708.529500:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5269 00000100:00080000:2.0:1761232708.529504:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232708.529508:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232708.529511:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232708.529518:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895078464/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232708.529524:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232708.529526:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232708.529527:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232708.529529:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5269 00000100:00080000:2.0:1761232708.529530:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232708.529532:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232708.529535:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232708.529540:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895078528/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232708.529545:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232708.529546:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232708.529547:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232708.529549:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5269 00000100:00080000:2.0:1761232708.529550:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232708.529552:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232708.529555:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232708.529561:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895078592/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232713.649366:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232713.649387:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232713.649390:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232713.649400:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5274 00000100:00080000:2.0:1761232713.649403:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232713.649409:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232713.649441:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895078656/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232713.649460:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232713.649462:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232713.649463:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232713.649466:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5274 00000100:00080000:2.0:1761232713.649467:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232713.649469:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232713.649476:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232713.649485:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895078720/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232713.649491:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232713.649493:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232713.649494:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232713.649498:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5274 00000100:00080000:2.0:1761232713.649499:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232713.649501:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232713.649503:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232713.649510:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895078784/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232713.649517:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232713.649522:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232713.649523:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232713.649527:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5274 00000100:00080000:2.0:1761232713.649529:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232713.649531:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232713.649534:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232713.649542:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895078848/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232713.649547:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232713.649550:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232713.649551:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232713.649553:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5274 00000100:00080000:2.0:1761232713.649554:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232713.649557:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232713.649559:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232713.649565:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895078912/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232713.649570:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232713.649572:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232713.649572:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232713.649574:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5274 00000100:00080000:2.0:1761232713.649576:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232713.649581:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232713.649585:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232713.649593:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895078976/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232713.649605:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232713.649611:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232713.649613:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232713.649618:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5274 00000100:00080000:2.0:1761232713.649619:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232713.649623:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232713.649626:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232713.649636:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895079040/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232718.769480:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895079104:192.168.201.157@tcp:250:kworker/u8:2.0 00000800:00000100:2.0:1761232718.769567:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232718.769597:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232718/real 1761232718] req@000000007aee79ba x1846780895079104/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232743 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232718.769638:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895079104/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232743 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232718.769654:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232718.769658:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232718.769664:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895079104:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761232718.769694:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895079168:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232718.769710:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232718.769715:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232718/real 1761232718] req@000000009706a782 x1846780895079168/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232743 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232718.769727:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895079168/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232743 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232718.769733:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232718.769734:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232718.769736:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895079168:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232718.769739:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895079232:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232718.769750:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232718.769753:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232718/real 1761232718] req@000000003b40830d x1846780895079232/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232743 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232718.769759:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895079232/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232743 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232718.769764:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232718.769766:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232718.769767:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895079232:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232718.769770:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895079296:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232718.769780:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232718.769783:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232718/real 1761232718] req@00000000c9993cd4 x1846780895079296/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232743 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232718.769789:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895079296/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232743 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232718.769793:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232718.769795:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232718.769797:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895079296:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232718.769800:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895079360:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232718.769811:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232718.769814:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232718/real 1761232718] req@000000002625ec0b x1846780895079360/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232743 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232718.769820:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895079360/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232743 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232718.769825:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232718.769827:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232718.769828:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895079360:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232718.769831:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895079424:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232718.769840:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232718.769844:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232718/real 1761232718] req@00000000bbf6af7b x1846780895079424/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232743 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232718.769849:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895079424/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232743 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232718.769854:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232718.769856:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232718.769858:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895079424:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232718.769861:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895079488:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232718.769871:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232718.769875:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232718/real 1761232718] req@00000000829a5595 x1846780895079488/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232743 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232718.769880:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895079488/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232743 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232718.769885:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232718.769887:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232718.769889:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895079488:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232723.889552:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895079552:192.168.201.157@tcp:250:kworker/u8:2.0 00000800:00000100:0.0:1761232723.889652:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232723.889676:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232723/real 1761232723] req@0000000057e55ffa x1846780895079552/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232748 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232723.889697:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895079552/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232748 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232723.889718:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232723.889723:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232723.889730:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895079552:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761232723.889756:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895079616:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761232723.889774:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232723.889779:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232723/real 1761232723] req@0000000089b8c4f1 x1846780895079616/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232748 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232723.889787:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895079616/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232748 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232723.889791:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232723.889793:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232723.889795:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895079616:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761232723.889800:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895079680:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232723.889811:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232723.889814:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232723/real 1761232723] req@000000002a0ba65f x1846780895079680/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232748 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232723.889821:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895079680/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232748 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232723.889825:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232723.889826:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232723.889827:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895079680:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232723.889830:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895079744:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232723.889839:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232723.889842:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232723/real 1761232723] req@00000000829a5595 x1846780895079744/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232748 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232723.889848:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895079744/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232748 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232723.889852:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232723.889853:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232723.889855:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895079744:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232723.889858:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895079808:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761232723.889867:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232723.889871:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232723/real 1761232723] req@00000000bbf6af7b x1846780895079808/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232748 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232723.889877:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895079808/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232748 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232723.889881:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232723.889883:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232723.889884:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895079808:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761232723.889887:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895079872:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232723.889896:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232723.889900:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232723/real 1761232723] req@000000002625ec0b x1846780895079872/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232748 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232723.889905:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895079872/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232748 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232723.889909:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232723.889910:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232723.889912:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895079872:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232723.889925:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895079936:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232723.889934:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232723.889938:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232723/real 1761232723] req@00000000c9993cd4 x1846780895079936/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232748 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232723.889943:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895079936/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232748 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232723.889948:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232723.889950:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232723.889951:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895079936:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00080000:2.0:1761232729.009434:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232729.009454:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232729.009461:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232729.009482:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5289 00000100:00080000:2.0:1761232729.009489:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232729.009502:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232729.009567:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895080000/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232729.009621:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232729.009628:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232729.009631:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232729.009638:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5289 00000100:00080000:2.0:1761232729.009642:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232729.009649:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232729.009667:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232729.009699:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895080064/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761232729.009701:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895080000:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761232729.009740:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232729.009747:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232729.009750:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232729.009760:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5289 00000100:00080000:2.0:1761232729.009764:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232729.009771:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232729.009782:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1761232729.009790:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232729.009813:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895080128/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761232729.009818:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232729/real 1761232729] req@000000004f29ca26 x1846780895080000/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232754 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232729.009904:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1761232729.009905:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895080000/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232754 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232729.009930:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232729.009934:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232729.009937:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232729.009939:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232729.009947:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895080000:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761232729.009954:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5289 00000100:00080000:2.0:1761232729.009960:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232729.009968:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:0.0:1761232729.009975:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895080064:192.168.201.157@tcp:38:kworker/u8:3.0 00000008:00100000:2.0:1761232729.009979:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1761232729.010005:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232729.010018:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895080192/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761232729.010019:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232729/real 1761232729] req@0000000033765045 x1846780895080064/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232754 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232729.010061:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895080064/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232754 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232729.010083:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232729.010088:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761232729.010091:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1761232729.010093:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895080064:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761232729.010106:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895080128:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761232729.010108:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232729.010113:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232729.010126:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5289 00000800:00000100:0.0:1761232729.010134:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761232729.010136:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1761232729.010142:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232729/real 1761232729] req@000000009d315fa5 x1846780895080128/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232754 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232729.010152:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:0.0:1761232729.010163:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895080128/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232754 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:2.0:1761232729.010170:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1761232729.010172:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232729.010177:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232729.010183:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895080128:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761232729.010242:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895080192:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232729.010268:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232729.010280:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232729/real 1761232729] req@0000000040d5f361 x1846780895080192/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232754 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232729.010296:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895080192/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232754 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232729.010304:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232729.010309:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232729.010313:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895080192:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232729.010331:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895080256/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232729.010370:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232729.010378:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232729.010383:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232729.010397:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5289 00000100:00080000:2.0:1761232729.010402:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232729.010412:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:0.0:1761232729.010415:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895080256:192.168.201.157@tcp:38:kworker/u8:3.0 00000008:00100000:2.0:1761232729.010434:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1761232729.010447:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232729.010456:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232729/real 1761232729] req@000000007dcf84ae x1846780895080256/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232754 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232729.010466:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895080320/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761232729.010468:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895080256/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232754 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232729.010502:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232729.010513:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232729.010519:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895080256:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:2.0:1761232729.010538:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1761232729.010543:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895080320:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761232729.010556:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232729.010562:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1761232729.010568:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761232729.010574:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5289 00000100:00100000:0.0:1761232729.010578:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232729/real 1761232729] req@000000007091f547 x1846780895080320/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232754 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232729.010587:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1761232729.010607:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895080320/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232754 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232729.010608:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:0.0:1761232729.010622:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232729.010626:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000008:00100000:2.0:1761232729.010627:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232729.010636:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895080320:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232729.010661:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895080384/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761232729.010729:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895080384:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761232729.010751:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232729.010757:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232729/real 1761232729] req@0000000064a2cea0 x1846780895080384/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232754 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761232729.010764:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895080384/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232754 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232729.010769:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232729.010771:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232729.010773:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895080384:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1761232734.129312:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232734.129321:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232734.129325:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232734.129336:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5295 00000100:00080000:0.0:1761232734.129339:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232734.129346:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232734.129380:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895080448/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232734.129403:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232734.129406:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232734.129407:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232734.129410:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5295 00000100:00080000:0.0:1761232734.129411:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232734.129414:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232734.129422:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232734.129431:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895080512/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232734.129438:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232734.129440:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232734.129442:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232734.129444:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5295 00000100:00080000:0.0:1761232734.129446:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232734.129448:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232734.129452:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232734.129459:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895080576/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232734.129465:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232734.129468:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232734.129468:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232734.129471:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5295 00000100:00080000:0.0:1761232734.129472:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232734.129478:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232734.129481:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232734.129490:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895080640/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232734.129495:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232734.129498:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232734.129499:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232734.129502:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5295 00000100:00080000:0.0:1761232734.129503:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232734.129505:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232734.129508:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232734.129516:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895080704/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232734.129521:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232734.129524:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232734.129525:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232734.129527:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5295 00000100:00080000:0.0:1761232734.129529:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232734.129531:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232734.129535:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232734.129541:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895080768/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232734.129545:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232734.129548:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232734.129549:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232734.129551:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5295 00000100:00080000:0.0:1761232734.129552:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232734.129554:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232734.129557:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232734.129565:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895080832/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232739.249391:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232739.249406:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232739.249411:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232739.249426:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5300 00000100:00080000:0.0:1761232739.249432:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232739.249443:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232739.249491:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895080896/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232739.249527:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232739.249532:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232739.249535:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232739.249539:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5300 00000100:00080000:0.0:1761232739.249542:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232739.249546:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232739.249558:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232739.249572:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895080960/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232739.249585:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232739.249590:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232739.249595:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232739.249610:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5300 00000100:00080000:0.0:1761232739.249613:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232739.249617:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232739.249623:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232739.249642:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895081024/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232739.249653:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232739.249657:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232739.249659:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232739.249663:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5300 00000100:00080000:0.0:1761232739.249667:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232739.249670:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232739.249674:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232739.249686:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895081088/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232739.249698:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232739.249705:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232739.249710:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232739.249717:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5300 00000100:00080000:0.0:1761232739.249720:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232739.249726:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232739.249732:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232739.249749:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895081152/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232739.249785:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232739.249790:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232739.249792:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232739.249804:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5300 00000100:00080000:0.0:1761232739.249811:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232739.249819:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232739.249827:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232739.249850:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895081216/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232739.249865:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232739.249869:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232739.249872:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232739.249876:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5300 00000100:00080000:0.0:1761232739.249878:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232739.249884:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232739.249894:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232739.249917:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895081280/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232744.369285:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232744.369296:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232744.369301:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232744.369312:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5305 00000100:00080000:0.0:1761232744.369317:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232744.369324:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232744.369356:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895081344/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232744.369379:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232744.369383:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232744.369384:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232744.369388:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5305 00000100:00080000:0.0:1761232744.369390:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232744.369393:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232744.369402:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232744.369414:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895081408/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232744.369423:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232744.369425:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232744.369427:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232744.369431:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5305 00000100:00080000:0.0:1761232744.369433:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232744.369435:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232744.369440:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232744.369448:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895081472/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232744.369455:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232744.369459:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232744.369460:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232744.369463:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5305 00000100:00080000:0.0:1761232744.369465:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232744.369468:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232744.369471:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232744.369479:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895081536/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232744.369485:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232744.369489:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232744.369490:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232744.369493:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5305 00000100:00080000:0.0:1761232744.369494:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232744.369498:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232744.369501:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232744.369509:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895081600/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232744.369522:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232744.369525:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232744.369527:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232744.369530:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5305 00000100:00080000:0.0:1761232744.369532:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232744.369536:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232744.369540:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232744.369548:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895081664/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232744.369555:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232744.369558:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232744.369559:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232744.369563:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5305 00000100:00080000:0.0:1761232744.369565:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232744.369568:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232744.369572:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232744.369581:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895081728/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232749.489320:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232749.489329:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232749.489332:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232749.489342:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5310 00000100:00080000:2.0:1761232749.489345:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232749.489351:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232749.489380:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895081792/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232749.489401:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232749.489403:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232749.489404:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232749.489406:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5310 00000100:00080000:2.0:1761232749.489408:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232749.489410:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232749.489417:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232749.489425:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895081856/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232749.489432:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232749.489436:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232749.489438:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232749.489445:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5310 00000100:00080000:2.0:1761232749.489446:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232749.489448:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232749.489451:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232749.489460:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895081920/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232749.489465:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232749.489468:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232749.489469:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232749.489471:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5310 00000100:00080000:2.0:1761232749.489472:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232749.489474:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232749.489477:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232749.489483:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895081984/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232749.489487:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232749.489489:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232749.489490:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232749.489492:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5310 00000100:00080000:2.0:1761232749.489493:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232749.489498:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232749.489502:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232749.489510:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895082048/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232749.489516:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232749.489518:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232749.489521:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232749.489527:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5310 00000100:00080000:2.0:1761232749.489530:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232749.489535:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232749.489539:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232749.489548:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895082112/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232749.489564:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232749.489573:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232749.489576:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232749.489580:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5310 00000100:00080000:2.0:1761232749.489583:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232749.489586:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232749.489591:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232749.489604:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895082176/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232754.609366:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232754.609377:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232754.609381:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232754.609393:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5315 00000100:00080000:2.0:1761232754.609398:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232754.609407:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232754.609452:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895082240/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232754.609481:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232754.609485:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232754.609487:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232754.609492:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5315 00000100:00080000:2.0:1761232754.609495:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232754.609499:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232754.609508:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232754.609519:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895082304/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232754.609530:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232754.609534:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232754.609536:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232754.609540:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5315 00000100:00080000:2.0:1761232754.609542:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232754.609545:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232754.609551:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232754.609565:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895082368/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232754.609585:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232754.609588:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232754.609589:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232754.609593:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5315 00000100:00080000:2.0:1761232754.609595:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232754.609599:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232754.609603:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232754.609614:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895082432/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232754.609630:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232754.609633:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232754.609635:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232754.609638:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5315 00000100:00080000:2.0:1761232754.609640:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232754.609644:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232754.609648:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232754.609662:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895082496/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232754.609690:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232754.609695:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232754.609697:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232754.609704:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5315 00000100:00080000:2.0:1761232754.609706:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232754.609715:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232754.609727:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232754.609746:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895082560/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232754.609766:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232754.609778:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232754.609782:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232754.609792:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5315 00000100:00080000:2.0:1761232754.609801:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232754.609813:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232754.609820:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232754.609841:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895082624/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232759.729302:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232759.729311:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232759.729315:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232759.729324:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5320 00000100:00080000:2.0:1761232759.729328:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232759.729333:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232759.729365:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895082688/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232759.729386:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232759.729389:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232759.729390:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232759.729393:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5320 00000100:00080000:2.0:1761232759.729395:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232759.729398:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232759.729404:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232759.729411:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895082752/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232759.729416:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232759.729418:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232759.729418:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232759.729422:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5320 00000100:00080000:2.0:1761232759.729423:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232759.729425:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232759.729428:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232759.729432:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895082816/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232759.729437:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232759.729438:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232759.729439:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232759.729442:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5320 00000100:00080000:2.0:1761232759.729444:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232759.729447:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232759.729449:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232759.729456:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895082880/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232759.729460:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232759.729463:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232759.729463:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232759.729466:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5320 00000100:00080000:2.0:1761232759.729467:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232759.729470:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232759.729473:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232759.729479:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895082944/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232759.729489:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232759.729492:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232759.729493:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232759.729499:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5320 00000100:00080000:2.0:1761232759.729502:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232759.729509:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232759.729515:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232759.729521:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895083008/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232759.729527:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232759.729529:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232759.729531:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232759.729534:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5320 00000100:00080000:2.0:1761232759.729536:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232759.729538:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232759.729541:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232759.729547:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895083072/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232764.849289:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232764.849301:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232764.849305:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232764.849320:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5325 00000100:00080000:2.0:1761232764.849325:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232764.849332:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232764.849371:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895083136/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232764.849399:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232764.849403:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232764.849405:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232764.849409:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5325 00000100:00080000:2.0:1761232764.849412:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232764.849416:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232764.849425:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232764.849435:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895083136:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761232764.849439:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895083200/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232764.849448:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232764.849450:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232764.849451:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232764.849454:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5325 00000100:00080000:2.0:1761232764.849456:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232764.849458:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232764.849461:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232764.849467:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895083264/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232764.849473:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232764.849475:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232764.849476:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232764.849480:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5325 00000100:00080000:2.0:1761232764.849481:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232764.849483:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232764.849485:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232764.849492:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895083328/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232764.849497:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232764.849499:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232764.849501:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1761232764.849502:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761232764.849506:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5325 00000100:00080000:2.0:1761232764.849508:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232764.849511:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232764.849515:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232764.849521:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895083392/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232764.849527:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1761232764.849528:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232764/real 1761232764] req@0000000099b7e5d9 x1846780895083136/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232789 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232764.849531:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232764.849533:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232764.849540:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5325 00000100:00080000:2.0:1761232764.849544:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232764.849548:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232764.849550:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232764.849557:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895083456/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761232764.849561:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895083136/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232789 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232764.849592:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232764.849593:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232764.849597:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761232764.849598:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232764.849600:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1761232764.849603:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895083136:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761232764.849605:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5325 00000100:00080000:2.0:1761232764.849610:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232764.849613:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232764.849616:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232764.849622:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895083520/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761232764.849632:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895083200:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761232764.849652:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232764.849658:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232764/real 1761232764] req@00000000f4e415d3 x1846780895083200/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232789 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232764.849666:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895083200/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232789 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232764.849670:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232764.849672:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232764.849674:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895083200:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761232764.849677:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895083264:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232764.849688:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232764.849692:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232764/real 1761232764] req@000000004ae5bc9b x1846780895083264/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232789 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232764.849715:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895083264/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232789 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232764.849724:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232764.849726:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232764.849729:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895083264:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232764.849736:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895083328:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232764.849757:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232764.849764:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232764/real 1761232764] req@000000008ebccab5 x1846780895083328/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232789 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232764.849776:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895083328/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232789 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232764.849785:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232764.849787:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232764.849790:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895083328:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232764.849854:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895083392:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761232764.849868:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232764.849873:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232764/real 1761232764] req@00000000a1e8c6bb x1846780895083392/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232789 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232764.849880:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895083392/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232789 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232764.849885:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232764.849886:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232764.849888:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895083392:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761232764.849891:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895083456:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232764.849925:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232764.849933:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232764/real 1761232764] req@00000000c10cebe6 x1846780895083456/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232789 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232764.849945:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895083456/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232789 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232764.849954:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232764.849956:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232764.849959:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895083456:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761232764.849964:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895083520:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761232764.849981:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232764.849988:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232764/real 1761232764] req@00000000ac7d8d5b x1846780895083520/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232789 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232764.850000:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895083520/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232789 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232764.850008:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232764.850011:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232764.850013:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895083520:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00080000:0.0:1761232769.969410:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232769.969426:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232769.969433:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232769.969452:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5330 00000100:00080000:0.0:1761232769.969458:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232769.969470:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232769.969533:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895083584/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232769.969586:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232769.969591:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232769.969593:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232769.969598:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5330 00000100:00080000:0.0:1761232769.969601:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232769.969605:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232769.969618:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232769.969633:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895083648/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232769.969643:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232769.969647:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232769.969649:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232769.969653:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5330 00000100:00080000:0.0:1761232769.969655:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232769.969658:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232769.969663:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232769.969672:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895083712/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232769.969680:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232769.969684:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232769.969687:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232769.969691:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5330 00000100:00080000:0.0:1761232769.969694:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232769.969699:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232769.969705:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232769.969715:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895083776/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232769.969729:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232769.969763:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232769.969769:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232769.969783:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5330 00000100:00080000:0.0:1761232769.969787:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232769.969791:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232769.969797:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232769.969812:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895083840/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232769.969823:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232769.969827:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232769.969830:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232769.969833:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5330 00000100:00080000:0.0:1761232769.969836:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232769.969841:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232769.969846:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232769.969859:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895083904/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232769.969867:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232769.969870:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232769.969874:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232769.969879:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5330 00000100:00080000:0.0:1761232769.969882:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232769.969886:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232769.969892:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232769.969910:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895083968/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761232775.089941:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232775.089944:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232775.089946:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895084224:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232775.089950:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895084288:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761232775.089973:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232775.089978:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232775/real 1761232775] req@0000000080131ea7 x1846780895084288/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232800 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232775.089986:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895084288/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232800 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232775.089991:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232775.089993:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232775.089995:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895084288:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232775.089999:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895084352:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232775.090010:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232775.090015:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232775/real 1761232775] req@000000007adf99ab x1846780895084352/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232800 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232775.090022:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895084352/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232800 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232775.090027:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232775.090029:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232775.090031:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895084352:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232775.090035:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895084416:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232775.090057:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232775.090062:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232775/real 1761232775] req@00000000e99e5a12 x1846780895084416/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232800 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232775.090070:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895084416/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232800 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232775.090076:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232775.090077:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232775.090079:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895084416:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232775.857439:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232778.929408:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232779.953358:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1761232780.209400:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895084480:192.168.201.157@tcp:250:kworker/u8:3.0 00000800:00000100:1.0:1761232780.209466:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232780.209489:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232780/real 1761232780] req@0000000070fa3bd6 x1846780895084480/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232805 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232780.209514:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895084480/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232805 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232780.209528:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232780.209533:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232780.209539:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895084480:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761232780.209546:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895084544:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761232780.209561:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232780.209566:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232780/real 1761232780] req@0000000090702bc4 x1846780895084544/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232805 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232780.209573:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895084544/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232805 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232780.209578:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232780.209579:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232780.209581:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895084544:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232780.209600:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895084608:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232780.209611:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232780.209615:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232780/real 1761232780] req@00000000b6fdd7f9 x1846780895084608/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232805 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232780.209639:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895084608/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232805 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232780.209645:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232780.209646:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232780.209648:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895084608:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232780.209651:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895084672:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232780.209665:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232780.209669:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232780/real 1761232780] req@00000000e99e5a12 x1846780895084672/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232805 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232780.209675:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895084672/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232805 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232780.209680:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232780.209682:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232780.209683:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895084672:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232780.209687:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895084736:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761232780.209697:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232780.209701:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232780/real 1761232780] req@000000007adf99ab x1846780895084736/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232805 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232780.209706:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895084736/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232805 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232780.209711:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232780.209713:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232780.209714:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895084736:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232780.209717:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895084800:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232780.209726:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232780.209730:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232780/real 1761232780] req@0000000080131ea7 x1846780895084800/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232805 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232780.209735:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895084800/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232805 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232780.209739:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232780.209740:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232780.209741:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895084800:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232780.209744:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895084864:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232780.209759:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232780.209766:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232780/real 1761232780] req@000000008e58c4b8 x1846780895084864/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232805 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232780.209776:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895084864/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232805 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232780.209783:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232780.209785:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232780.209788:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895084864:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232783.025475:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232784.049494:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1761232785.329617:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895084928:192.168.201.157@tcp:250:kworker/u8:2.0 00000800:00000100:2.0:1761232785.329733:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232785.329773:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232785/real 1761232785] req@000000007aee79ba x1846780895084928/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232810 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232785.329808:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895084928/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232810 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232785.329840:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232785.329847:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232785.329858:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895084928:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761232785.329903:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895084992:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232785.329949:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232785.329964:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232785/real 1761232785] req@000000009706a782 x1846780895084992/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232810 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232785.329988:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895084992/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232810 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232785.329998:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232785.330001:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232785.330004:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895084992:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232785.330011:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895085056:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232785.330032:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232785.330040:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232785/real 1761232785] req@000000003b40830d x1846780895085056/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232810 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232785.330051:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895085056/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232810 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232785.330059:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232785.330062:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232785.330065:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895085056:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232785.330071:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895085120:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232785.330088:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232785.330095:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232785/real 1761232785] req@00000000c9993cd4 x1846780895085120/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232810 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232785.330106:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895085120/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232810 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232785.330114:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232785.330118:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232785.330121:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895085120:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232785.330127:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895085184:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232785.330145:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232785.330152:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232785/real 1761232785] req@000000002625ec0b x1846780895085184/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232810 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232785.330163:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895085184/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232810 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232785.330171:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232785.330174:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232785.330177:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895085184:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232785.330183:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895085248:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232785.330232:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232785.330243:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232785/real 1761232785] req@00000000bbf6af7b x1846780895085248/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232810 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232785.330255:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895085248/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232810 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232785.330263:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232785.330266:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232785.330269:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895085248:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232785.330297:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895085312:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232785.330324:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232785.330333:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232785/real 1761232785] req@00000000829a5595 x1846780895085312/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232810 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232785.330345:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895085312/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232810 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232785.330353:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232785.330355:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232785.330358:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895085312:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761232787.121504:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232788.145492:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761232790.449380:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232790.449393:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232790.449399:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232790.449414:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5351 00000100:00080000:0.0:1761232790.449419:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232790.449428:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232790.449473:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000addc3db8 x1846780895085376/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232790.449556:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232790.449561:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232790.449563:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232790.449567:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5351 00000100:00080000:0.0:1761232790.449570:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232790.449574:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:2.0:1761232790.449580:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895085376:192.168.201.157@tcp:250:kworker/u8:2.0 00000008:00100000:0.0:1761232790.449584:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232790.449601:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000078e7052d x1846780895085440/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232790.449610:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232790.449615:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232790.449616:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232790.449620:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5351 00000100:00080000:0.0:1761232790.449622:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232790.449625:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232790.449629:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232790.449639:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895085504/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232790.449646:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232790.449648:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232790.449650:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1761232790.449656:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761232790.449657:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5351 00000100:00080000:0.0:1761232790.449661:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232790.449664:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232790.449668:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232790.449679:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895085568/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232790.449683:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232790/real 1761232790] req@00000000addc3db8 x1846780895085376/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232815 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232790.449704:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232790.449709:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232790.449711:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232790.449717:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5351 00000100:00100000:2.0:1761232790.449720:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000addc3db8 x1846780895085376/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232815 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232790.449722:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232790.449730:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232790.449738:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1761232790.449743:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232790.449748:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232790.449749:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895085632/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232790.449757:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895085376:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761232790.449764:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232790.449768:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232790.449770:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232790.449775:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5351 00000100:00080000:0.0:1761232790.449777:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232790.449780:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232790.449782:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232790.449792:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895085440:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761232790.449793:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895085696/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232790.449809:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232790.449812:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232790.449814:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232790.449817:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5351 00000100:00080000:0.0:1761232790.449819:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232790.449823:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:2.0:1761232790.449828:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:0.0:1761232790.449830:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232790.449841:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232790/real 1761232790] req@0000000078e7052d x1846780895085440/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232815 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232790.449845:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895085760/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232790.449878:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000078e7052d x1846780895085440/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232815 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232790.449888:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232790.449891:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232790.449894:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895085440:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232790.449901:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895085504:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232790.449917:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232790.449922:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232790/real 1761232790] req@000000005beaae31 x1846780895085504/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232815 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232790.449932:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895085504/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232815 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232790.449938:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232790.449939:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232790.449942:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895085504:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232790.449947:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895085568:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232790.449960:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232790.449964:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232790/real 1761232790] req@0000000034d49f59 x1846780895085568/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232815 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232790.449973:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895085568/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232815 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232790.449979:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232790.449980:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232790.449982:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895085568:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232790.449987:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895085632:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232790.450000:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232790.450005:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232790/real 1761232790] req@00000000eca9ddc2 x1846780895085632/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232815 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232790.450014:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895085632/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232815 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232790.450020:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232790.450021:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232790.450023:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895085632:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232790.450041:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895085696:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232790.450053:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232790.450058:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232790/real 1761232790] req@000000006f2efc8b x1846780895085696/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232815 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232790.450066:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895085696/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232815 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232790.450072:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232790.450074:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232790.450076:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895085696:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232790.450080:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895085760:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232790.450091:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232790.450096:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232790/real 1761232790] req@000000004862b808 x1846780895085760/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232815 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232790.450110:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895085760/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232815 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232790.450120:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232790.450123:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232790.450127:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895085760:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761232791.217546:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232792.241378:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232795.313476:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761232795.569338:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232795.569348:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232795.569352:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232795.569362:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5356 00000100:00080000:0.0:1761232795.569366:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232795.569372:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232795.569406:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895085824/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232795.569425:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232795.569428:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232795.569429:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232795.569432:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5356 00000100:00080000:0.0:1761232795.569433:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232795.569436:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232795.569455:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232795.569465:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895085888/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232795.569471:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895085824:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761232795.569475:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232795.569479:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232795.569480:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232795.569483:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5356 00000100:00080000:0.0:1761232795.569485:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232795.569486:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232795.569490:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232795.569498:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895085952/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232795.569505:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232795.569507:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232795.569509:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232795.569511:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5356 00000100:00080000:0.0:1761232795.569513:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232795.569516:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232795.569519:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232795.569526:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895086016/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232795.569531:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232795.569533:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232795.569534:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1761232795.569538:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761232795.569539:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5356 00000100:00080000:0.0:1761232795.569542:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232795.569545:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232795.569548:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232795.569556:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895086080/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232795.569561:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232795/real 1761232795] req@00000000c6e06344 x1846780895085824/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232820 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232795.569564:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232795.569571:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232795.569573:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232795.569578:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5356 00000100:00080000:0.0:1761232795.569580:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1761232795.569584:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895085824/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232820 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232795.569585:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232795.569590:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1761232795.569598:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1761232795.569599:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895086144/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232795.569602:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232795.569609:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895085824:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761232795.569611:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232795.569614:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232795.569616:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232795.569619:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5356 00000100:00080000:0.0:1761232795.569620:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232795.569622:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232795.569624:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232795.569630:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895086208/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232795.569634:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895085888:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232795.569656:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232795.569663:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232795/real 1761232795] req@000000005593c994 x1846780895085888/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232820 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232795.569670:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895085888/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232820 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232795.569675:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232795.569677:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232795.569679:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895085888:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232795.569682:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895085952:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232795.569692:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232795.569696:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232795/real 1761232795] req@000000007bd1ec64 x1846780895085952/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232820 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232795.569702:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895085952/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232820 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232795.569706:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232795.569707:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232795.569709:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895085952:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232795.569712:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895086016:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232795.569721:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232795.569725:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232795/real 1761232795] req@000000001db96463 x1846780895086016/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232820 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232795.569730:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895086016/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232820 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232795.569735:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232795.569736:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232795.569737:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895086016:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232795.569740:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895086080:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232795.569752:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232795.569756:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232795/real 1761232795] req@00000000efa29eb9 x1846780895086080/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232820 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232795.569761:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895086080/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232820 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232795.569765:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232795.569767:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232795.569768:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895086080:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232795.569771:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895086144:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232795.569780:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232795.569783:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232795/real 1761232795] req@000000006d61dd06 x1846780895086144/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232820 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232795.569789:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895086144/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232820 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232795.569793:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232795.569794:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232795.569796:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895086144:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232795.569811:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895086208:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232795.569820:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232795.569824:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232795/real 1761232795] req@00000000ef655201 x1846780895086208/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232820 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232795.569831:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895086208/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232820 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232795.569836:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232795.569837:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232795.569839:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895086208:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761232796.337472:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232799.409481:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232800.433552:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761232800.689428:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761232800.689439:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761232800.689443:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761232800.689454:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5361 00000100:00080000:1.0:1761232800.689458:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761232800.689464:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761232800.689507:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000880f16b4 x1846780895086272/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761232800.689549:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761232800.689554:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761232800.689557:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761232800.689562:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5361 00000100:00080000:1.0:1761232800.689565:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761232800.689571:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761232800.689581:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232800.689587:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895086272:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761232800.689597:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000053e52331 x1846780895086336/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761232800.689607:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761232800.689610:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761232800.689612:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761232800.689615:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5361 00000100:00080000:1.0:1761232800.689617:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761232800.689619:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761232800.689624:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232800.689632:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090ac513e x1846780895086400/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761232800.689639:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761232800.689641:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761232800.689642:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761232800.689646:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5361 00000100:00080000:1.0:1761232800.689648:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761232800.689652:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761232800.689656:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1761232800.689662:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232800.689666:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d66930c2 x1846780895086464/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761232800.689673:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761232800.689676:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761232800.689679:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761232800.689683:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5361 00000100:00080000:1.0:1761232800.689684:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761232800.689692:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:2.0:1761232800.689692:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232800/real 1761232800] req@00000000880f16b4 x1846780895086272/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232825 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:1.0:1761232800.689699:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232800.689711:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ffe86d04 x1846780895086528/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232800.689717:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000880f16b4 x1846780895086272/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232825 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232800.689725:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761232800.689733:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761232800.689738:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232800.689741:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232800.689745:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5361 00000100:00080000:2.0:1761232800.689748:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:1.0:1761232800.689750:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1761232800.689755:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895086272:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761232800.689757:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761232800.689765:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232800.689775:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df5b2270 x1846780895086592/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761232800.689781:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761232800.689784:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761232800.689786:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1761232800.689789:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895086336:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:1.0:1761232800.689793:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5361 00000100:00080000:1.0:1761232800.689797:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761232800.689800:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761232800.689805:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232800.689814:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df0b94a8 x1846780895086656/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:2.0:1761232800.689821:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232800.689835:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232800/real 1761232800] req@0000000053e52331 x1846780895086336/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232825 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232800.689847:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000053e52331 x1846780895086336/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232825 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232800.689854:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232800.689856:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232800.689859:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895086336:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232800.689865:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895086400:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232800.689879:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232800.689884:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232800/real 1761232800] req@0000000090ac513e x1846780895086400/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232825 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232800.689893:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090ac513e x1846780895086400/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232825 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232800.689898:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232800.689900:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232800.689902:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895086400:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232800.689905:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895086464:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232800.689917:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232800.689922:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232800/real 1761232800] req@00000000d66930c2 x1846780895086464/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232825 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232800.689929:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d66930c2 x1846780895086464/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232825 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232800.689935:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232800.689936:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232800.689938:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895086464:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232800.689942:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895086528:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761232800.689954:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232800.689959:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232800/real 1761232800] req@00000000ffe86d04 x1846780895086528/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232825 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232800.689968:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ffe86d04 x1846780895086528/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232825 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232800.689974:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232800.689975:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232800.689977:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895086528:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232800.689981:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895086592:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232800.689992:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232800.689997:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232800/real 1761232800] req@00000000df5b2270 x1846780895086592/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232825 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232800.690005:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df5b2270 x1846780895086592/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232825 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232800.690010:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232800.690011:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232800.690014:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895086592:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232800.690032:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895086656:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232800.690045:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232800.690050:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232800/real 1761232800] req@00000000df0b94a8 x1846780895086656/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232825 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232800.690058:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df0b94a8 x1846780895086656/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232825 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232800.690064:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232800.690066:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232800.690068:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895086656:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232803.505486:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232804.529510:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1761232805.809717:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895086720:192.168.201.157@tcp:250:kworker/u8:2.0 00000800:00000100:2.0:1761232805.809824:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232805.809861:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232805/real 1761232805] req@000000002a0ba65f x1846780895086720/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232830 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232805.809913:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895086720/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232830 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232805.809948:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232805.809955:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232805.809967:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895086720:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761232805.810036:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895086784:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232805.810074:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232805.810088:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232805/real 1761232805] req@0000000089b8c4f1 x1846780895086784/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232830 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232805.810102:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895086784/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232830 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232805.810111:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232805.810113:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232805.810117:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895086784:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232805.810123:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895086848:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232805.810145:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232805.810151:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232805/real 1761232805] req@0000000057e55ffa x1846780895086848/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232830 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232805.810162:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895086848/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232830 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232805.810170:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232805.810172:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232805.810175:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895086848:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232805.810181:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895086912:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232805.810197:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232805.810203:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232805/real 1761232805] req@00000000829a5595 x1846780895086912/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232830 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232805.810214:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895086912/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232830 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232805.810252:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232805.810254:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232805.810257:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895086912:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232805.810263:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895086976:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232805.810295:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232805.810303:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232805/real 1761232805] req@00000000bbf6af7b x1846780895086976/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232830 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232805.810315:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895086976/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232830 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232805.810380:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232805.810400:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232805.810406:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895086976:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232805.810416:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895087040:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232805.810450:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232805.810464:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232805/real 1761232805] req@000000002625ec0b x1846780895087040/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232830 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232805.810482:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895087040/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232830 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232805.810494:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232805.810496:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232805.810500:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895087040:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232805.810526:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895087104:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232805.810550:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232805.810558:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232805/real 1761232805] req@00000000c9993cd4 x1846780895087104/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232830 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232805.810587:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895087104/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232830 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232805.810600:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232805.810603:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232805.810606:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895087104:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761232807.601619:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232808.625502:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761232810.929417:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232810.929435:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232810.929443:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232810.929461:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5371 00000100:00080000:2.0:1761232810.929467:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232810.929478:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232810.929539:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895087168/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232810.929580:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232810.929585:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232810.929587:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232810.929592:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5371 00000100:00080000:2.0:1761232810.929595:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232810.929599:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232810.929612:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232810.929626:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895087232/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232810.929636:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232810.929640:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232810.929642:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232810.929646:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5371 00000100:00080000:2.0:1761232810.929648:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232810.929652:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232810.929656:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232810.929667:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895087296/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232810.929676:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232810.929681:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232810.929682:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232810.929686:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5371 00000100:00080000:2.0:1761232810.929688:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232810.929693:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232810.929698:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232810.929710:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895087360/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232810.929719:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232810.929724:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232810.929726:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232810.929730:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5371 00000100:00080000:2.0:1761232810.929732:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232810.929736:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232810.929742:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232810.929746:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895087168:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761232810.929757:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895087424/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232810.929770:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232810.929773:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232810.929775:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232810.929780:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5371 00000100:00080000:2.0:1761232810.929784:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232810.929788:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232810.929795:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232810.929810:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895087488/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232810.929819:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232810.929823:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232810.929825:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232810.929830:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5371 00000100:00080000:2.0:1761232810.929833:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232810.929836:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232810.929840:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232810.929856:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895087552/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232810.929899:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895087232:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232810.929932:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895087296:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232810.929958:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895087360:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232810.930002:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895087424:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232810.930031:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895087488:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232810.930057:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895087552:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232811.697577:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1761232814.001736:0:2208:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:0.0:1761232814.001814:0:2208:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1761232814.001835:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232814.001900:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232814.001915:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232814.001927:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232814.001940:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232814.001954:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232814.001969:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:1.0:1761232814.002046:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232810/real 1761232814] req@00000000fd4aa3a2 x1846780895087424/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232835 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232814.002108:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895087424/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232835 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232814.002133:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232814.002142:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232814.002150:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895087424:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232814.002165:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232810/real 1761232814] req@000000006c1041ba x1846780895087488/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232835 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232814.002191:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895087488/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232835 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232814.002289:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232814.002296:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232814.002305:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895087488:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232814.002322:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232810/real 1761232814] req@000000006d8c3eb0 x1846780895087552/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232835 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232814.002349:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895087552/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232835 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232814.002363:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232814.002368:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232814.002374:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895087552:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232814.002384:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232810/real 1761232814] req@0000000088939481 x1846780895087168/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232835 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232814.002403:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895087168/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232835 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232814.002427:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232814.002431:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232814.002436:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895087168:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761232814.002446:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232810/real 1761232814] req@000000006ff1fdbc x1846780895087232/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232835 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232814.002464:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895087232/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232835 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232814.002475:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232814.002480:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232814.002484:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895087232:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232814.002494:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232810/real 1761232814] req@00000000135e9a6e x1846780895087296/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232835 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232814.002511:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895087296/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232835 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232814.002523:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232814.002528:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232814.002532:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895087296:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232814.002543:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232810/real 1761232814] req@00000000fd3935a6 x1846780895087360/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232835 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232814.002569:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895087360/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232835 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232814.002595:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232814.002600:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232814.002606:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895087360:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232815.793505:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761232816.049362:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232816.049378:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232816.049384:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232816.049404:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5376 00000100:00080000:2.0:1761232816.049410:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232816.049421:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232816.049479:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895087616/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232816.049516:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232816.049521:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232816.049524:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232816.049529:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5376 00000100:00080000:2.0:1761232816.049533:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232816.049541:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232816.049556:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232816.049574:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895087680/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232816.049579:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895087616:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761232816.049594:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232816.049599:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232816.049601:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232816.049605:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5376 00000100:00080000:2.0:1761232816.049608:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232816.049613:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232816.049623:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232816.049635:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895087744/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232816.049645:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232816.049649:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232816.049651:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232816.049656:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5376 00000100:00080000:2.0:1761232816.049659:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232816.049662:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232816.049668:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232816.049683:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895087808/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761232816.049696:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761232816.049707:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232816.049713:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232816.049715:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232816.049723:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5376 00000100:00080000:2.0:1761232816.049725:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232816.049730:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232816.049735:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232816.049743:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232816/real 1761232816] req@000000004f29ca26 x1846780895087616/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232841 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232816.049754:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895087872/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232816.049816:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232816.049822:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232816.049824:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761232816.049828:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895087616/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232841 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232816.049839:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5376 00000100:00080000:2.0:1761232816.049851:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232816.049865:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:1.0:1761232816.049866:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232816.049875:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000008:00100000:2.0:1761232816.049878:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232816.049887:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895087616:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761232816.049912:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895087936/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232816.049934:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895087680:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:2.0:1761232816.049935:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232816.049959:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232816.049963:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232816.049974:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5376 00000800:00000100:1.0:1761232816.049982:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761232816.049984:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232816.049999:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761232816.050009:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232816/real 1761232816] req@0000000033765045 x1846780895087680/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232841 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:2.0:1761232816.050016:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232816.050040:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895087680/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232841 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232816.050052:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1761232816.050056:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895088000/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761232816.050059:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232816.050076:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895087680:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232816.050111:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895087744:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232816.050147:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232816.050163:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232816/real 1761232816] req@000000009d315fa5 x1846780895087744/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232841 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232816.050175:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895087744/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232841 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232816.050182:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232816.050184:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232816.050186:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895087744:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232816.050192:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895087808:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232816.050264:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232816.050274:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232816/real 1761232816] req@0000000040d5f361 x1846780895087808/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232841 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232816.050285:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895087808/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232841 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232816.050292:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232816.050295:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232816.050297:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895087808:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232816.050303:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895087872:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761232816.050326:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232816.050334:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232816/real 1761232816] req@000000007dcf84ae x1846780895087872/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232841 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232816.050346:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895087872/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232841 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232816.050354:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232816.050356:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232816.050359:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895087872:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232816.050389:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895087936:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232816.050411:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232816.050419:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232816/real 1761232816] req@000000007091f547 x1846780895087936/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232841 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232816.050430:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895087936/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232841 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232816.050437:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232816.050440:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232816.050442:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895087936:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232816.050448:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895088000:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232816.050467:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232816.050474:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232816/real 1761232816] req@0000000064a2cea0 x1846780895088000/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232841 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232816.050484:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895088000/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232841 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232816.050490:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232816.050494:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232816.050496:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895088000:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232819.889467:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761232821.169352:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232821.169368:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232821.169374:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232821.169390:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5382 00000100:00080000:2.0:1761232821.169394:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232821.169402:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232821.169460:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895088064/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232821.169492:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232821.169496:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232821.169497:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232821.169501:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5382 00000100:00080000:2.0:1761232821.169503:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232821.169506:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232821.169538:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232821.169554:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895088128/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232821.169569:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232821.169574:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232821.169575:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232821.169582:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5382 00000100:00080000:2.0:1761232821.169585:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232821.169589:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232821.169596:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232821.169607:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895088192/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232821.169620:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895088064:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761232821.169628:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232821.169641:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232821.169643:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232821.169650:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5382 00000100:00080000:2.0:1761232821.169652:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232821.169657:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232821.169663:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232821.169680:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895088256/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232821.169689:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232821.169694:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232821.169695:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232821.169703:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5382 00000100:00080000:2.0:1761232821.169706:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232821.169710:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232821.169716:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232821.169737:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895088320/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761232821.169749:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761232821.169772:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232821.169777:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232821.169780:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761232821.169788:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232821/real 1761232821] req@0000000064a2cea0 x1846780895088064/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232846 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232821.169790:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5382 00000100:00080000:2.0:1761232821.169801:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232821.169815:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232821.169830:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232821.169859:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895088064/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232846 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232821.169860:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895088384/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232821.169940:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232821.169947:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761232821.169948:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232821.169951:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761232821.169955:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761232821.169962:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5382 00000100:00100000:1.0:1761232821.169969:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895088064:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761232821.169972:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232821.169982:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761232821.169991:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895088128:192.168.201.157@tcp:38:kworker/u8:3.0 00000008:00100000:2.0:1761232821.169992:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232821.170032:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895088448/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761232821.170052:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232821.170068:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232821/real 1761232821] req@000000007091f547 x1846780895088128/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232846 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232821.170089:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895088128/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232846 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232821.170099:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232821.170102:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232821.170105:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895088128:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232821.170181:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895088192:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232821.170275:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232821.170292:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232821/real 1761232821] req@000000007dcf84ae x1846780895088192/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232846 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232821.170315:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895088192/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232846 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232821.170328:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232821.170331:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232821.170335:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895088192:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232821.170343:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895088256:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232821.170370:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232821.170379:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232821/real 1761232821] req@0000000040d5f361 x1846780895088256/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232846 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232821.170391:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895088256/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232846 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232821.170400:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232821.170402:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232821.170406:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895088256:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232821.170412:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895088320:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761232821.170430:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232821.170438:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232821/real 1761232821] req@000000009d315fa5 x1846780895088320/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232846 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232821.170449:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895088320/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232846 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232821.170458:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232821.170462:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232821.170465:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895088320:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232821.170470:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895088384:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232821.170488:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232821.170495:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232821/real 1761232821] req@0000000033765045 x1846780895088384/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232846 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232821.170506:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895088384/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232846 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232821.170513:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232821.170516:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232821.170519:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895088384:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232821.170526:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895088448:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232821.170543:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232821.170550:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232821/real 1761232821] req@000000004f29ca26 x1846780895088448/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232846 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232821.170561:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895088448/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232846 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232821.170569:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232821.170572:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232821.170575:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895088448:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232823.985419:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1761232826.289673:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895088512:192.168.201.157@tcp:250:kworker/u8:3.0 00000800:00000100:1.0:1761232826.289774:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232826.289810:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232826/real 1761232826] req@000000003b40830d x1846780895088512/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232851 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232826.289852:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895088512/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232851 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232826.289897:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232826.289907:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232826.289921:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895088512:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761232826.289947:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895088576:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761232826.289985:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232826.290001:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232826/real 1761232826] req@000000009706a782 x1846780895088576/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232851 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232826.290016:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895088576/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232851 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232826.290025:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232826.290027:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232826.290031:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895088576:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232826.290062:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895088640:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232826.290083:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232826.290090:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232826/real 1761232826] req@000000007aee79ba x1846780895088640/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232851 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232826.290129:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895088640/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232851 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232826.290138:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232826.290141:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232826.290144:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895088640:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232826.290150:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895088704:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232826.290171:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232826.290180:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232826/real 1761232826] req@000000008e58c4b8 x1846780895088704/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232851 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232826.290191:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895088704/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232851 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232826.290231:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232826.290234:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232826.290238:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895088704:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232826.290245:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895088768:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761232826.290268:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232826.290277:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232826/real 1761232826] req@0000000080131ea7 x1846780895088768/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232851 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232826.290289:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895088768/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232851 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232826.290298:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232826.290301:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232826.290304:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895088768:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232826.290310:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895088832:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232826.290328:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232826.290335:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232826/real 1761232826] req@000000007adf99ab x1846780895088832/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232851 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232826.290346:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895088832/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232851 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232826.290355:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232826.290357:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232826.290360:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895088832:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232826.290365:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895088896:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232826.290382:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232826.290389:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232826/real 1761232826] req@00000000e99e5a12 x1846780895088896/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232851 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232826.290399:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895088896/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232851 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232826.290408:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232826.290410:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232826.290413:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895088896:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232828.081537:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761232831.409300:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232831.409313:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232831.409317:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232831.409328:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5392 00000100:00080000:2.0:1761232831.409332:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232831.409341:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232831.409381:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895088960/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232831.409412:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232831.409416:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232831.409418:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232831.409421:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5392 00000100:00080000:2.0:1761232831.409424:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232831.409428:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232831.409437:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232831.409463:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895089024/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232831.409473:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232831.409476:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232831.409478:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232831.409482:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5392 00000100:00080000:2.0:1761232831.409484:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232831.409488:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232831.409495:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232831.409506:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895089088/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232831.409529:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895088960:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761232831.409541:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232831.409546:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232831.409547:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232831.409558:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5392 00000100:00080000:2.0:1761232831.409561:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232831.409565:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232831.409571:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232831.409584:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895089152/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232831.409594:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1761232831.409599:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761232831.409602:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232831.409605:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232831.409609:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5392 00000100:00080000:2.0:1761232831.409612:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232831.409616:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232831.409621:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232831.409623:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232831/real 1761232831] req@00000000fd3935a6 x1846780895088960/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232856 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232831.409633:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895089216/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232831.409640:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895088960/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232856 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232831.409644:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232831.409652:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232831.409654:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761232831.409660:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232831.409661:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5392 00000100:00080000:2.0:1761232831.409665:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761232831.409666:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761232831.409670:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761232831.409673:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895088960:192.168.201.157@tcp:250:kworker/u8:3.0 00000008:00100000:2.0:1761232831.409676:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232831.409682:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895089024:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232831.409688:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895089280/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232831.409700:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232831.409704:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1761232831.409706:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761232831.409706:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232831.409713:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5392 00000100:00080000:2.0:1761232831.409714:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761232831.409716:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232831/real 1761232831] req@00000000135e9a6e x1846780895089024/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232856 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232831.409721:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232831.409727:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232831.409733:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895089024/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232856 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232831.409738:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895089344/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761232831.409749:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232831.409754:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232831.409760:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895089024:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232831.409784:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895089088:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232831.409807:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232831.409812:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232831/real 1761232831] req@000000006ff1fdbc x1846780895089088/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232856 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232831.409819:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895089088/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232856 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232831.409825:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232831.409827:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232831.409829:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895089088:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232831.409832:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895089152:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232831.409843:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232831.409847:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232831/real 1761232831] req@0000000088939481 x1846780895089152/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232856 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232831.409854:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895089152/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232856 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232831.409859:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232831.409860:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232831.409862:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895089152:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232831.409865:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895089216:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761232831.409875:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232831.409879:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232831/real 1761232831] req@000000006d8c3eb0 x1846780895089216/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232856 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232831.409886:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895089216/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232856 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232831.409891:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232831.409892:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232831.409894:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895089216:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232831.409898:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895089280:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232831.409909:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232831.409913:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232831/real 1761232831] req@000000006c1041ba x1846780895089280/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232856 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232831.409920:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895089280/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232856 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232831.409925:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232831.409926:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232831.409927:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895089280:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232831.409931:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895089344:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232831.409941:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232831.409945:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232831/real 1761232831] req@00000000fd4aa3a2 x1846780895089344/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232856 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232831.409951:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895089344/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232856 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232831.409956:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232831.409958:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232831.409960:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895089344:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232832.177395:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232836.273522:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761232836.529465:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232836.529482:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232836.529488:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232836.529504:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5397 00000100:00080000:0.0:1761232836.529509:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232836.529519:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232836.529579:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895089408/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232836.529617:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232836.529622:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232836.529624:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232836.529628:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5397 00000100:00080000:0.0:1761232836.529631:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232836.529635:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232836.529647:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232836.529664:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895089472/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232836.529676:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761232836.529677:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895089408:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761232836.529684:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232836.529687:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232836.529693:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5397 00000100:00080000:0.0:1761232836.529695:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232836.529698:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232836.529703:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232836.529715:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895089536/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232836.529723:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232836.529727:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232836.529728:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232836.529732:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5397 00000100:00080000:0.0:1761232836.529735:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232836.529739:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232836.529744:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232836.529755:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895089600/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232836.529762:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232836.529765:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232836.529766:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232836.529769:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5397 00000100:00080000:0.0:1761232836.529772:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232836.529776:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232836.529781:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761232836.529792:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232836.529795:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895089664/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232836.529806:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232836.529811:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232836.529813:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232836.529818:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5397 00000100:00080000:0.0:1761232836.529820:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232836.529824:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232836.529828:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232836.529834:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232836/real 1761232836] req@000000004130b6bd x1846780895089408/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232861 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232836.529841:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895089728/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232836.529871:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232836.529876:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232836.529878:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761232836.529883:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895089408/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232861 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232836.529886:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5397 00000100:00080000:0.0:1761232836.529892:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232836.529900:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232836.529908:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1761232836.529916:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1761232836.529923:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895089792/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761232836.529926:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232836.529938:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895089408:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761232836.530011:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895089472:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761232836.530041:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232836.530052:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232836/real 1761232836] req@000000001f9ec359 x1846780895089472/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232861 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232836.530064:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895089472/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232861 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232836.530072:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232836.530075:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232836.530078:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895089472:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761232836.530084:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895089536:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232836.530101:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232836.530108:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232836/real 1761232836] req@000000009d1c7f73 x1846780895089536/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232861 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232836.530118:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895089536/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232861 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232836.530133:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232836.530135:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232836.530137:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895089536:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232836.530142:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895089600:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232836.530158:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232836.530165:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232836/real 1761232836] req@00000000d67bb589 x1846780895089600/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232861 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232836.530174:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895089600/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232861 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232836.530182:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232836.530184:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232836.530187:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895089600:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232836.530192:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895089664:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761232836.530251:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232836.530265:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232836/real 1761232836] req@00000000a4a2c44a x1846780895089664/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232861 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232836.530284:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895089664/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232861 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232836.530297:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232836.530301:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232836.530306:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895089664:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761232836.530316:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895089728:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232836.530341:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232836.530350:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232836/real 1761232836] req@000000004a74da19 x1846780895089728/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232861 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232836.530361:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895089728/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232861 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232836.530369:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232836.530371:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232836.530374:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895089728:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232836.530379:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895089792:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232836.530395:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232836.530401:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232836/real 1761232836] req@000000002d1b8ef7 x1846780895089792/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232861 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232836.530411:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895089792/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232861 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232836.530418:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232836.530420:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232836.530423:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895089792:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761232840.369449:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1761232841.649761:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895089856:192.168.201.157@tcp:250:kworker/u8:2.0 00000800:00000100:1.0:1761232841.649892:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232841.649982:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232841/real 1761232841] req@00000000b6fdd7f9 x1846780895089856/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232866 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232841.650065:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895089856/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232866 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232841.650116:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232841.650126:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232841.650147:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895089856:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761232841.650262:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895089920:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761232841.650307:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232841.650323:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232841/real 1761232841] req@0000000090702bc4 x1846780895089920/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232866 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232841.650339:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895089920/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232866 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232841.650349:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232841.650353:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232841.650357:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895089920:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761232841.650364:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895089984:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232841.650386:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232841.650394:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232841/real 1761232841] req@0000000070fa3bd6 x1846780895089984/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232866 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232841.650406:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895089984/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232866 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232841.650423:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232841.650425:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232841.650428:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895089984:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232841.650434:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895090048:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232841.650452:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232841.650459:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232841/real 1761232841] req@00000000e99e5a12 x1846780895090048/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232866 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232841.650470:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895090048/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232866 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232841.650479:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232841.650481:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232841.650485:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895090048:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232841.650491:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895090112:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761232841.650509:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232841.650516:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232841/real 1761232841] req@000000007adf99ab x1846780895090112/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232866 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232841.650528:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895090112/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232866 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232841.650536:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232841.650539:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232841.650542:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895090112:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761232841.650549:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895090176:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232841.650567:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232841.650574:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232841/real 1761232841] req@0000000080131ea7 x1846780895090176/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232866 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232841.650586:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895090176/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232866 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232841.650594:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232841.650596:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232841.650599:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895090176:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232841.650626:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895090240:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232841.650644:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232841.650651:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232841/real 1761232841] req@000000008e58c4b8 x1846780895090240/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232866 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232841.650662:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895090240/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232866 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232841.650670:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232841.650673:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232841.650676:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895090240:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761232844.465509:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1761232846.769622:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895090304:192.168.201.157@tcp:250:kworker/u8:2.0 00000800:00000100:2.0:1761232846.769725:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232846.769765:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232846/real 1761232846] req@000000007aee79ba x1846780895090304/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232871 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232846.769805:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895090304/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232871 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232846.769843:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232846.769854:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232846.769916:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895090304:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761232846.769935:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895090368:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232846.769984:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232846.769998:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232846/real 1761232846] req@000000009706a782 x1846780895090368/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232871 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232846.770029:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895090368/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232871 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232846.770039:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232846.770044:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232846.770048:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895090368:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232846.770082:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895090432:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232846.770104:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232846.770112:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232846/real 1761232846] req@000000003b40830d x1846780895090432/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232871 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232846.770124:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895090432/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232871 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232846.770133:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232846.770135:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232846.770138:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895090432:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232846.770148:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895090496:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232846.770165:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232846.770173:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232846/real 1761232846] req@00000000c9993cd4 x1846780895090496/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232871 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232846.770184:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895090496/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232871 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232846.770193:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232846.770196:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232846.770236:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895090496:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232846.770244:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895090560:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232846.770269:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232846.770278:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232846/real 1761232846] req@000000002625ec0b x1846780895090560/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232871 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232846.770291:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895090560/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232871 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232846.770299:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232846.770302:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232846.770305:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895090560:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232846.770311:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895090624:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232846.770328:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232846.770336:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232846/real 1761232846] req@00000000bbf6af7b x1846780895090624/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232871 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232846.770346:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895090624/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232871 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232846.770355:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232846.770357:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232846.770360:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895090624:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232846.770366:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895090688:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232846.770386:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232846.770393:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232846/real 1761232846] req@00000000829a5595 x1846780895090688/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232871 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232846.770404:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895090688/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232871 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232846.770412:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232846.770415:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232846.770418:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895090688:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761232848.561430:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761232851.889417:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232851.889430:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232851.889435:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232851.889448:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5412 00000100:00080000:0.0:1761232851.889454:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232851.889462:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232851.889507:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895090752/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232851.889540:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232851.889544:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232851.889546:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232851.889551:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5412 00000100:00080000:0.0:1761232851.889553:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232851.889557:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232851.889567:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232851.889579:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895090816/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232851.889598:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232851.889602:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232851.889604:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232851.889608:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5412 00000100:00080000:0.0:1761232851.889610:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232851.889613:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232851.889618:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232851.889629:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895090880/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232851.889640:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232851.889643:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232851.889645:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232851.889650:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5412 00000100:00080000:0.0:1761232851.889652:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232851.889656:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232851.889660:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232851.889670:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895090944/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232851.889677:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895090752:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761232851.889688:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232851.889693:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232851.889695:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232851.889700:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5412 00000100:00080000:0.0:1761232851.889703:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232851.889707:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232851.889713:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232851.889725:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895091008/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232851.889734:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232851.889738:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232851.889739:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232851.889744:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5412 00000100:00080000:0.0:1761232851.889745:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232851.889749:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232851.889752:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232851.889766:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895091072/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232851.889777:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1761232851.889779:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761232851.889782:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232851.889787:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232851.889792:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5412 00000100:00080000:0.0:1761232851.889795:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232851.889800:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232851.889837:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232851.889854:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895091136/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232851.889859:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232851/real 1761232851] req@000000005d83493f x1846780895090752/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232876 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232851.889902:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005d83493f x1846780895090752/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232876 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232851.889916:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232851.889920:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232851.889924:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895090752:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761232851.889933:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895090816:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232851.889960:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232851.889970:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232851/real 1761232851] req@00000000f3ed2588 x1846780895090816/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232876 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232851.889982:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f3ed2588 x1846780895090816/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232876 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232851.889991:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232851.889994:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232851.889997:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895090816:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232851.890003:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895090880:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232851.890021:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232851.890028:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232851/real 1761232851] req@000000002d1b8ef7 x1846780895090880/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232876 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232851.890038:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895090880/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232876 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232851.890046:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232851.890048:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232851.890051:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895090880:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232851.890082:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895090944:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232851.890100:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232851.890107:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232851/real 1761232851] req@000000004a74da19 x1846780895090944/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232876 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232851.890118:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895090944/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232876 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232851.890125:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232851.890127:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232851.890130:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895090944:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232851.890137:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895091008:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232851.890154:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232851.890161:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232851/real 1761232851] req@00000000a4a2c44a x1846780895091008/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232876 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232851.890172:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895091008/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232876 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232851.890180:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232851.890182:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232851.890185:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895091008:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232851.890190:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895091072:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232851.890253:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232851.890263:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232851/real 1761232851] req@00000000d67bb589 x1846780895091072/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232876 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232851.890275:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895091072/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232876 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232851.890282:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232851.890284:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232851.890288:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895091072:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232851.890293:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895091136:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232851.890312:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232851.890319:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232851/real 1761232851] req@000000009d1c7f73 x1846780895091136/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232876 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232851.890331:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895091136/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232876 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232851.890339:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232851.890341:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232851.890343:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895091136:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761232852.657441:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232856.753596:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1761232857.009667:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895091200:192.168.201.157@tcp:250:kworker/u8:3.0 00000800:00000100:2.0:1761232857.009807:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232857.009839:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232857/real 1761232857] req@0000000057e55ffa x1846780895091200/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232882 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232857.009873:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895091200/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232882 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232857.009900:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232857.009906:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232857.009914:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895091200:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761232857.009929:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895091264:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761232857.009967:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232857.009983:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232857/real 1761232857] req@0000000089b8c4f1 x1846780895091264/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232882 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232857.009996:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895091264/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232882 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232857.010005:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232857.010009:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232857.010012:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895091264:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232857.010042:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895091328:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232857.010060:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232857.010067:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232857/real 1761232857] req@000000002a0ba65f x1846780895091328/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232882 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232857.010077:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895091328/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232882 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232857.010083:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232857.010085:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232857.010088:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895091328:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232857.010093:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895091392:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232857.010106:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232857.010112:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232857/real 1761232857] req@00000000829a5595 x1846780895091392/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232882 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232857.010122:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895091392/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232882 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232857.010128:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232857.010130:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232857.010132:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895091392:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232857.010137:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895091456:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761232857.010154:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232857.010160:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232857/real 1761232857] req@00000000bbf6af7b x1846780895091456/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232882 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232857.010170:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895091456/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232882 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232857.010177:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232857.010179:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232857.010181:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895091456:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232857.010185:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895091520:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232857.010199:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232857.010205:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232857/real 1761232857] req@000000002625ec0b x1846780895091520/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232882 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232857.010214:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895091520/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232882 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232857.010258:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232857.010260:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232857.010264:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895091520:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232857.010269:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895091584:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232857.010289:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232857.010297:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232857/real 1761232857] req@00000000c9993cd4 x1846780895091584/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232882 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232857.010307:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895091584/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232882 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232857.010315:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232857.010317:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232857.010320:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895091584:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232860.849423:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1761232862.129462:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895091648:192.168.201.157@tcp:250:kworker/u8:2.0 00000800:00000100:2.0:1761232862.129531:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232862.129553:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232862/real 1761232862] req@000000003b40830d x1846780895091648/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232887 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232862.129566:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895091648/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232887 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232862.129573:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232862.129575:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232862.129578:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895091648:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761232862.129582:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895091712:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232862.129596:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232862.129601:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232862/real 1761232862] req@000000009706a782 x1846780895091712/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232887 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232862.129607:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895091712/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232887 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232862.129611:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232862.129613:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232862.129614:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895091712:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232862.129618:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895091776:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232862.129629:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232862.129633:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232862/real 1761232862] req@000000007aee79ba x1846780895091776/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232887 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232862.129639:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895091776/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232887 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232862.129643:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232862.129644:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232862.129646:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895091776:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232862.129666:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895091840:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232862.129676:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232862.129680:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232862/real 1761232862] req@000000008e58c4b8 x1846780895091840/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232887 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232862.129712:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895091840/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232887 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232862.129717:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232862.129718:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232862.129720:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895091840:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232862.129723:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895091904:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232862.129735:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232862.129739:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232862/real 1761232862] req@0000000080131ea7 x1846780895091904/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232887 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232862.129746:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895091904/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232887 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232862.129750:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232862.129751:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232862.129753:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895091904:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232862.129756:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895091968:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232862.129765:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232862.129769:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232862/real 1761232862] req@000000007adf99ab x1846780895091968/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232887 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232862.129775:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895091968/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232887 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232862.129780:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232862.129781:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232862.129782:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895091968:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232862.129785:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895092032:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232862.129794:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232862.129797:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232862/real 1761232862] req@00000000e99e5a12 x1846780895092032/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232887 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232862.129803:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895092032/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232887 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232862.129806:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232862.129809:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232862.129810:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895092032:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761232864.945473:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761232867.249473:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232867.249489:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232867.249497:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232867.249511:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5428 00000100:00080000:0.0:1761232867.249516:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232867.249525:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232867.249574:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895092096/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232867.249622:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232867.249649:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232867.249652:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232867.249659:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5428 00000100:00080000:0.0:1761232867.249663:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232867.249670:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232867.249685:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232867.249710:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895092160/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232867.249730:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232867.249734:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232867.249736:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232867.249742:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5428 00000100:00080000:0.0:1761232867.249745:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232867.249749:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232867.249757:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232867.249771:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895092096:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761232867.249776:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895092224/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232867.249794:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232867.249798:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232867.249800:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232867.249804:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5428 00000100:00080000:0.0:1761232867.249808:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232867.249811:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232867.249816:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232867.249830:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895092288/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232867.249841:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232867.249845:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232867.249846:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232867.249850:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5428 00000100:00080000:0.0:1761232867.249852:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232867.249858:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232867.249863:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1761232867.249878:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232867.249880:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895092352/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232867.249892:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232867.249897:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232867.249899:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232867.249905:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5428 00000100:00080000:0.0:1761232867.249909:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1761232867.249916:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232867/real 1761232867] req@000000001f9ec359 x1846780895092096/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232892 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232867.249917:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232867.249926:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232867.249942:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895092416/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232867.249953:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1761232867.249955:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895092096/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232892 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232867.249959:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232867.249963:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232867.249975:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5428 00000100:00080000:0.0:1761232867.249980:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232867.249989:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:2.0:1761232867.249992:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1761232867.249995:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1761232867.250003:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232867.250013:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895092480/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232867.250015:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895092096:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761232867.250039:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895092160:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232867.250074:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232867.250083:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232867/real 1761232867] req@000000004130b6bd x1846780895092160/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232892 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232867.250096:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895092160/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232892 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232867.250106:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232867.250109:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232867.250112:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895092160:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232867.250143:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895092224:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232867.250167:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232867.250175:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232867/real 1761232867] req@00000000ef655201 x1846780895092224/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232892 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232867.250187:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895092224/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232892 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232867.250196:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232867.250241:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232867.250246:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895092224:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232867.250253:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895092288:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232867.250277:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232867.250286:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232867/real 1761232867] req@000000006d61dd06 x1846780895092288/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232892 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232867.250299:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895092288/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232892 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232867.250308:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232867.250310:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232867.250313:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895092288:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232867.250319:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895092352:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232867.250337:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232867.250345:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232867/real 1761232867] req@00000000efa29eb9 x1846780895092352/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232892 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232867.250356:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895092352/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232892 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232867.250366:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232867.250368:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232867.250372:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895092352:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232867.250378:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895092416:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232867.250394:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232867.250402:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232867/real 1761232867] req@000000001db96463 x1846780895092416/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232892 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232867.250412:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895092416/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232892 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232867.250420:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232867.250423:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232867.250426:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895092416:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232867.250431:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895092480:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232867.250448:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232867.250455:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232867/real 1761232867] req@000000007bd1ec64 x1846780895092480/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232892 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232867.250466:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895092480/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232892 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232867.250475:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232867.250477:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232867.250479:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895092480:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761232869.041425:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1761232872.369671:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895092544:192.168.201.157@tcp:250:kworker/u8:3.0 00000800:00000100:2.0:1761232872.369775:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232872.369814:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232872/real 1761232872] req@0000000070fa3bd6 x1846780895092544/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232897 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232872.369858:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895092544/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232897 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232872.369893:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232872.369904:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232872.369917:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895092544:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761232872.369936:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895092608:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761232872.369975:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232872.369988:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232872/real 1761232872] req@0000000090702bc4 x1846780895092608/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232897 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232872.370002:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895092608/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232897 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232872.370011:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232872.370015:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232872.370018:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895092608:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232872.370050:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895092672:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232872.370069:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232872.370077:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232872/real 1761232872] req@00000000b6fdd7f9 x1846780895092672/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232897 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232872.370098:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895092672/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232897 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232872.370107:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232872.370109:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232872.370112:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895092672:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232872.370118:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895092736:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232872.370135:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232872.370143:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232872/real 1761232872] req@00000000e99e5a12 x1846780895092736/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232897 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232872.370154:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895092736/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232897 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232872.370162:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232872.370164:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232872.370167:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895092736:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232872.370173:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895092800:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761232872.370190:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232872.370197:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232872/real 1761232872] req@000000007adf99ab x1846780895092800/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232897 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232872.370269:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895092800/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232897 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232872.370280:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232872.370286:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232872.370291:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895092800:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232872.370302:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895092864:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232872.370341:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232872.370356:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232872/real 1761232872] req@0000000080131ea7 x1846780895092864/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232897 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232872.370374:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895092864/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232897 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232872.370389:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232872.370393:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232872.370398:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895092864:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232872.370408:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895092928:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232872.370463:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232872.370476:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232872/real 1761232872] req@000000008e58c4b8 x1846780895092928/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232897 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232872.370490:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895092928/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232897 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232872.370499:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232872.370501:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232872.370505:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895092928:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232873.137539:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232877.233445:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761232877.489378:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232877.489392:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232877.489396:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232877.489410:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5438 00000100:00080000:2.0:1761232877.489414:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232877.489422:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232877.489467:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895092992/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232877.489496:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232877.489499:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232877.489500:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232877.489503:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5438 00000100:00080000:2.0:1761232877.489505:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232877.489509:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232877.489529:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232877.489540:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895093056/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232877.489548:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895092992:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761232877.489551:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232877.489562:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232877.489564:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232877.489567:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5438 00000100:00080000:2.0:1761232877.489569:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232877.489571:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232877.489575:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232877.489586:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895093120/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232877.489593:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232877.489596:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232877.489597:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232877.489600:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5438 00000100:00080000:2.0:1761232877.489601:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232877.489605:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232877.489608:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232877.489614:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895093184/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232877.489619:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232877.489622:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232877.489624:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232877.489627:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5438 00000100:00080000:2.0:1761232877.489629:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232877.489631:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232877.489633:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232877.489641:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895093248/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232877.489647:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232877.489650:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232877.489652:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761232877.489656:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895093056:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:2.0:1761232877.489657:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5438 00000100:00080000:2.0:1761232877.489666:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232877.489673:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232877.489677:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232877.489687:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895093312/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232877.489694:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895093120:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761232877.489696:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232877.489706:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232877.489709:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232877.489714:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5438 00000100:00080000:2.0:1761232877.489716:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232877.489719:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761232877.489724:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895093184:192.168.201.157@tcp:8:kworker/u8:3.0 00000008:00100000:2.0:1761232877.489724:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232877.489753:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895093376/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232877.489760:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895093248:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232877.489791:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895093312:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232877.489812:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895093376:192.168.201.157@tcp:8:kworker/u8:3.0 00000400:00000100:0.0:1761232880.561547:0:2209:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:0.0:1761232880.561599:0:2209:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1761232880.561615:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232880.561654:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232880.561662:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232880.561670:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232880.561677:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232880.561685:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761232880.561693:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:1.0:1761232880.561736:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232877/real 1761232880] req@00000000f4e415d3 x1846780895093312/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232902 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232880.561768:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895093312/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232902 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232880.561779:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232880.561783:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232880.561788:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895093312:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232880.561795:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232877/real 1761232880] req@0000000099b7e5d9 x1846780895093376/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232902 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232880.561813:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895093376/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232902 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232880.561822:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232880.561824:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232880.561827:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895093376:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232880.561832:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232877/real 1761232880] req@00000000c10cebe6 x1846780895093056/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232902 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232880.561842:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895093056/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232902 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232880.561850:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232880.561853:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232880.561856:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895093056:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232880.561861:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232877/real 1761232880] req@00000000a1e8c6bb x1846780895093120/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232902 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232880.561872:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895093120/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232902 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232880.561878:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232880.561881:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232880.561883:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895093120:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232880.561888:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232877/real 1761232880] req@000000008ebccab5 x1846780895093184/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232902 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232880.561903:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895093184/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232902 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232880.561910:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232880.561912:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232880.561915:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895093184:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232880.561920:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232877/real 1761232880] req@000000004ae5bc9b x1846780895093248/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232902 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232880.561930:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895093248/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232902 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232880.561937:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232880.561940:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232880.561943:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895093248:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232880.561949:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232877/real 1761232880] req@00000000ac7d8d5b x1846780895092992/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232902 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232880.561960:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895092992/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232902 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232880.561967:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232880.561970:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232880.561973:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895092992:192.168.201.157@tcp:250:kworker/u8:3.0 00010000:00010000:1.0:1761232881.329388:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761232882.609384:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232882.609394:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232882.609397:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232882.609418:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5443 00000100:00080000:0.0:1761232882.609421:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232882.609428:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232882.609468:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895093440/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232882.609498:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232882.609501:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232882.609502:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232882.609504:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5443 00000100:00080000:0.0:1761232882.609506:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232882.609508:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232882.609514:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232882.609522:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895093504/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232882.609528:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232882.609531:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232882.609532:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232882.609534:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5443 00000100:00080000:0.0:1761232882.609536:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232882.609538:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232882.609541:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232882.609546:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895093568/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232882.609551:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895093440:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761232882.609554:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232882.609561:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232882.609563:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232882.609566:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5443 00000100:00080000:0.0:1761232882.609568:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232882.609569:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232882.609572:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232882.609582:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895093632/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232882.609586:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232882.609589:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232882.609590:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232882.609592:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5443 00000100:00080000:0.0:1761232882.609593:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232882.609596:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232882.609599:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232882.609604:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895093696/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232882.609609:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232882.609612:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232882.609612:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232882.609615:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5443 00000100:00080000:0.0:1761232882.609616:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1761232882.609620:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761232882.609621:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232882.609626:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232882.609633:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895093760/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232882.609642:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232882/real 1761232882] req@000000005593c994 x1846780895093440/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232907 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232882.609647:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232882.609652:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232882.609654:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232882.609661:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5443 00000100:00080000:0.0:1761232882.609664:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761232882.609669:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895093440/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232907 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232882.609670:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232882.609677:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1761232882.609688:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232882.609692:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232882.609694:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895093824/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232882.609699:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895093440:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761232882.609712:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895093504:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761232882.609738:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232882.609746:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232882/real 1761232882] req@00000000c6e06344 x1846780895093504/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232907 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232882.609758:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895093504/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232907 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232882.609764:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232882.609766:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232882.609770:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895093504:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761232882.609797:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895093568:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232882.609821:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232882.609829:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232882/real 1761232882] req@000000004862b808 x1846780895093568/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232907 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232882.609839:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895093568/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232907 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232882.609847:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232882.609848:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232882.609851:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895093568:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232882.609857:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895093632:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232882.609876:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232882.609883:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232882/real 1761232882] req@000000006f2efc8b x1846780895093632/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232907 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232882.609893:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895093632/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232907 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232882.609900:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232882.609902:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232882.609906:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895093632:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232882.609911:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895093696:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761232882.609930:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232882.609997:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232882/real 1761232882] req@00000000eca9ddc2 x1846780895093696/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232907 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232882.610010:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895093696/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232907 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232882.610017:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232882.610020:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232882.610022:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895093696:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761232882.610028:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895093760:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232882.610052:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232882.610059:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232882/real 1761232882] req@0000000034d49f59 x1846780895093760/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232907 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232882.610069:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895093760/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232907 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232882.610076:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232882.610078:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232882.610080:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895093760:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232882.610086:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895093824:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232882.610105:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232882.610113:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232882/real 1761232882] req@000000005beaae31 x1846780895093824/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232907 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232882.610123:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895093824/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232907 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232882.610130:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232882.610133:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232882.610136:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895093824:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761232885.425538:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761232887.729296:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232887.729307:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232887.729310:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232887.729320:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5448 00000100:00080000:0.0:1761232887.729323:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232887.729330:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232887.729365:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000078e7052d x1846780895093888/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232887.729383:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232887.729386:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232887.729388:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232887.729390:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5448 00000100:00080000:0.0:1761232887.729391:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232887.729408:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232887.729418:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232887.729426:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895093888:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761232887.729430:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000addc3db8 x1846780895093952/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232887.729439:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232887.729441:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232887.729442:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232887.729445:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5448 00000100:00080000:0.0:1761232887.729447:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232887.729449:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232887.729452:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232887.729460:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895094016/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232887.729465:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232887.729467:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232887.729468:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232887.729470:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5448 00000100:00080000:0.0:1761232887.729472:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232887.729475:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232887.729478:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232887.729489:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895094080/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761232887.729489:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761232887.729499:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232887.729503:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232887.729504:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232887.729508:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5448 00000100:00100000:1.0:1761232887.729511:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232887/real 1761232887] req@0000000078e7052d x1846780895093888/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232912 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232887.729513:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232887.729520:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232887.729526:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232887.729528:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000078e7052d x1846780895093888/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232912 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232887.729537:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1761232887.729538:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895094144/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761232887.729540:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232887.729547:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895093888:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761232887.729558:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232887.729561:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232887.729563:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232887.729568:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5448 00000100:00080000:0.0:1761232887.729570:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761232887.729572:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895093952:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:0.0:1761232887.729577:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232887.729582:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761232887.729593:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232887.729596:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895094208/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232887.729601:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232887/real 1761232887] req@00000000addc3db8 x1846780895093952/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232912 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232887.729617:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761232887.729620:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000addc3db8 x1846780895093952/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232912 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232887.729623:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232887.729625:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761232887.729631:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761232887.729634:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5448 00000100:00080000:1.0:1761232887.729635:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761232887.729639:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761232887.729641:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895093952:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:0.0:1761232887.729646:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761232887.729649:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895094016:192.168.201.157@tcp:8:kworker/u8:2.0 00000008:00100000:0.0:1761232887.729653:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761232887.729666:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232887.729669:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895094272/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232887.729674:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232887/real 1761232887] req@000000005beaae31 x1846780895094016/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232912 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232887.729693:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895094016/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232912 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232887.729709:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232887.729710:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232887.729713:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895094016:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232887.729716:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895094080:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232887.729729:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232887.729733:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232887/real 1761232887] req@0000000034d49f59 x1846780895094080/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232912 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232887.729739:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895094080/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232912 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232887.729743:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232887.729744:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232887.729746:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895094080:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232887.729749:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895094144:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761232887.729758:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232887.729762:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232887/real 1761232887] req@00000000eca9ddc2 x1846780895094144/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232912 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232887.729767:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895094144/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232912 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232887.729771:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232887.729772:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232887.729774:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895094144:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761232887.729786:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895094208:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232887.729796:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232887.729800:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232887/real 1761232887] req@000000006f2efc8b x1846780895094208/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232912 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232887.729806:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895094208/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232912 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232887.729811:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232887.729812:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232887.729813:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895094208:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232887.729816:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895094272:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232887.729825:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232887.729829:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232887/real 1761232887] req@000000004862b808 x1846780895094272/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232912 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232887.729835:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895094272/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232912 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232887.729840:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232887.729841:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232887.729842:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895094272:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761232889.521367:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1761232892.849558:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895094336:192.168.201.157@tcp:250:kworker/u8:2.0 00000800:00000100:1.0:1761232892.849652:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232892.849680:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232892/real 1761232892] req@000000007aee79ba x1846780895094336/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232917 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232892.849695:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895094336/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232917 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232892.849704:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232892.849707:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232892.849710:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895094336:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761232892.849716:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895094400:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761232892.849731:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232892.849735:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232892/real 1761232892] req@000000009706a782 x1846780895094400/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232917 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232892.849742:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895094400/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232917 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232892.849747:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232892.849748:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232892.849750:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895094400:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761232892.849753:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895094464:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232892.849765:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232892.849771:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232892/real 1761232892] req@000000003b40830d x1846780895094464/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232917 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232892.849780:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895094464/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232917 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232892.849792:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232892.849795:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232892.849797:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895094464:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232892.849803:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895094528:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232892.849823:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232892.849830:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232892/real 1761232892] req@00000000c9993cd4 x1846780895094528/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232917 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232892.849839:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895094528/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232917 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232892.849846:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232892.849848:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232892.849851:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895094528:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232892.849857:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895094592:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761232892.849877:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232892.849885:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232892/real 1761232892] req@000000002625ec0b x1846780895094592/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232917 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232892.849896:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895094592/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232917 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232892.849903:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232892.849906:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232892.849909:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895094592:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761232892.849944:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895094656:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232892.849964:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232892.849972:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232892/real 1761232892] req@00000000bbf6af7b x1846780895094656/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232917 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232892.849983:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895094656/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232917 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232892.849988:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232892.849990:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232892.849992:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895094656:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232892.849995:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895094720:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232892.850010:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232892.850018:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232892/real 1761232892] req@00000000829a5595 x1846780895094720/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232917 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232892.850028:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895094720/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232917 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232892.850035:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232892.850038:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232892.850041:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895094720:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761232893.617459:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232897.713418:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761232897.969343:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232897.969352:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232897.969355:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232897.969364:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5458 00000100:00080000:2.0:1761232897.969367:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232897.969373:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232897.969406:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895094784/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232897.969427:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232897.969429:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232897.969430:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232897.969433:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5458 00000100:00080000:2.0:1761232897.969435:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232897.969437:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232897.969445:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232897.969453:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895094848/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232897.969459:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895094784:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761232897.969461:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232897.969468:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232897.969469:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232897.969472:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5458 00000100:00080000:2.0:1761232897.969473:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232897.969475:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232897.969478:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232897.969486:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895094912/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232897.969491:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232897.969493:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232897.969494:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232897.969495:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5458 00000100:00080000:2.0:1761232897.969497:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232897.969499:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232897.969502:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232897.969508:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895094976/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232897.969517:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232897.969520:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232897.969521:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232897.969524:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5458 00000100:00080000:2.0:1761232897.969525:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1761232897.969528:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761232897.969531:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232897.969534:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232897.969541:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895095040/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232897.969547:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232897.969549:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232897.969550:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761232897.969552:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232897/real 1761232897] req@00000000ac7d8d5b x1846780895094784/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232922 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232897.969556:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5458 00000100:00080000:2.0:1761232897.969560:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232897.969564:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232897.969568:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232897.969576:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895094784/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232922 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232897.969576:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895095104/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232897.969603:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232897.969606:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761232897.969609:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232897.969609:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761232897.969616:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761232897.969617:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5458 00000100:00080000:2.0:1761232897.969621:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761232897.969624:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895094784:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761232897.969625:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232897.969631:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232897.969638:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895095168/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232897.969651:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895094848:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761232897.969668:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232897.969673:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232897/real 1761232897] req@000000004ae5bc9b x1846780895094848/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232922 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232897.969680:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895094848/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232922 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232897.969685:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232897.969687:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232897.969689:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895094848:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232897.969692:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895094912:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232897.969702:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232897.969706:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232897/real 1761232897] req@000000008ebccab5 x1846780895094912/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232922 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232897.969712:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895094912/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232922 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232897.969720:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232897.969721:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232897.969723:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895094912:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232897.969726:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895094976:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232897.969735:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232897.969739:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232897/real 1761232897] req@00000000a1e8c6bb x1846780895094976/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232922 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232897.969745:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895094976/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232922 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232897.969749:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232897.969751:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232897.969752:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895094976:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232897.969755:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895095040:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761232897.969764:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232897.969768:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232897/real 1761232897] req@00000000c10cebe6 x1846780895095040/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232922 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232897.969774:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895095040/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232922 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232897.969778:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232897.969779:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232897.969781:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895095040:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232897.969784:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895095104:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232897.969793:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232897.969797:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232897/real 1761232897] req@0000000099b7e5d9 x1846780895095104/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232922 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232897.969803:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895095104/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232922 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232897.969807:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232897.969809:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232897.969810:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895095104:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232897.969813:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895095168:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232897.969822:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232897.969825:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232897/real 1761232897] req@00000000f4e415d3 x1846780895095168/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232922 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232897.969831:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895095168/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232922 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232897.969835:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232897.969836:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232897.969838:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895095168:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232901.809459:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761232903.089306:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232903.089319:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232903.089323:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232903.089336:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5464 00000100:00080000:2.0:1761232903.089339:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232903.089347:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232903.089383:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895095232/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232903.089412:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232903.089416:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232903.089418:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232903.089422:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5464 00000100:00080000:2.0:1761232903.089424:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232903.089428:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232903.089437:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232903.089451:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895095296/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232903.089462:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761232903.089464:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895095232:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761232903.089466:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232903.089469:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232903.089476:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5464 00000100:00080000:2.0:1761232903.089479:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232903.089482:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232903.089487:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232903.089501:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895095360/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232903.089509:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232903.089511:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232903.089513:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232903.089516:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5464 00000100:00080000:2.0:1761232903.089518:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232903.089522:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232903.089526:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761232903.089530:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232903.089540:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895095424/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232903.089550:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761232903.089552:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232903/real 1761232903] req@00000000fd4aa3a2 x1846780895095232/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232928 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232903.089557:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232903.089560:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232903.089567:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5464 00000100:00080000:2.0:1761232903.089570:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761232903.089571:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895095232/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232928 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232903.089578:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:1.0:1761232903.089583:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1761232903.089585:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1761232903.089587:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232903.089592:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895095232:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761232903.089600:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895095488/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232903.089614:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895095296:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:2.0:1761232903.089622:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232903.089627:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232903.089629:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1761232903.089633:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761232903.089636:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5464 00000100:00080000:2.0:1761232903.089640:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761232903.089643:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232903/real 1761232903] req@000000006c1041ba x1846780895095296/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232928 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232903.089646:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232903.089655:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232903.089658:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895095296/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232928 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232903.089665:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232903.089667:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232903.089672:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895095296:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232903.089673:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895095552/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232903.089679:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895095360:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761232903.089693:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232903.089696:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232903.089698:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1761232903.089699:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761232903.089707:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5464 00000100:00100000:1.0:1761232903.089709:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232903/real 1761232903] req@000000006d8c3eb0 x1846780895095360/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232928 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232903.089712:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232903.089719:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761232903.089722:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895095360/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232928 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:2.0:1761232903.089726:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1761232903.089736:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232903.089737:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232903.089739:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895095360:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232903.089744:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895095616/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232903.089746:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895095424:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232903.089764:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232903.089770:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232903/real 1761232903] req@0000000088939481 x1846780895095424/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232928 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232903.089779:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895095424/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232928 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232903.089783:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232903.089785:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232903.089787:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895095424:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232903.089799:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895095488:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761232903.089809:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232903.089812:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232903/real 1761232903] req@000000006ff1fdbc x1846780895095488/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232928 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232903.089819:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895095488/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232928 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232903.089823:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232903.089825:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232903.089826:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895095488:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232903.089830:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895095552:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232903.089839:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232903.089843:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232903/real 1761232903] req@00000000135e9a6e x1846780895095552/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232928 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232903.089849:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895095552/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232928 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232903.089853:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232903.089854:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232903.089856:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895095552:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232903.089859:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895095616:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232903.089867:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232903.089871:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232903/real 1761232903] req@00000000fd3935a6 x1846780895095616/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232928 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232903.089876:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895095616/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232928 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232903.089881:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232903.089882:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232903.089884:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895095616:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232905.905429:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1761232908.209521:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895095680:192.168.201.157@tcp:250:kworker/u8:2.0 00000800:00000100:2.0:1761232908.209603:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232908.209630:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232908/real 1761232908] req@000000002a0ba65f x1846780895095680/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232933 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232908.209650:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895095680/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232933 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232908.209663:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232908.209667:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232908.209672:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895095680:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761232908.209701:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895095744:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232908.209720:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232908.209727:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232908/real 1761232908] req@0000000089b8c4f1 x1846780895095744/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232933 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232908.209738:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895095744/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232933 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232908.209756:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232908.209760:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232908.209765:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895095744:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232908.209770:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895095808:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232908.209786:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232908.209795:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232908/real 1761232908] req@0000000057e55ffa x1846780895095808/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232933 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232908.209806:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895095808/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232933 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232908.209812:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232908.209814:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232908.209817:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895095808:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232908.209825:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895095872:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232908.209846:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232908.209850:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232908/real 1761232908] req@00000000829a5595 x1846780895095872/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232933 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232908.209857:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895095872/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232933 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232908.209863:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232908.209865:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232908.209867:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895095872:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232908.209870:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895095936:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232908.209883:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232908.209887:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232908/real 1761232908] req@00000000bbf6af7b x1846780895095936/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232933 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232908.209893:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895095936/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232933 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232908.209898:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232908.209899:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232908.209901:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895095936:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232908.209912:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895096000:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232908.209921:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232908.209925:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232908/real 1761232908] req@000000002625ec0b x1846780895096000/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232933 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232908.209932:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895096000/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232933 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232908.209936:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232908.209938:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232908.209939:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895096000:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232908.209943:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895096064:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232908.209951:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232908.209955:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232908/real 1761232908] req@00000000c9993cd4 x1846780895096064/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232933 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232908.209962:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895096064/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232933 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232908.209966:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232908.209967:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232908.209969:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895096064:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761232910.001351:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761232913.329326:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232913.329337:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232913.329341:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232913.329352:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5474 00000100:00080000:0.0:1761232913.329356:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232913.329363:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232913.329397:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895096128/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232913.329425:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232913.329430:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232913.329432:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232913.329436:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5474 00000100:00080000:0.0:1761232913.329439:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232913.329444:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232913.329454:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232913.329458:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895096128:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761232913.329469:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895096192/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232913.329481:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232913.329485:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232913.329487:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232913.329491:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5474 00000100:00080000:0.0:1761232913.329493:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232913.329497:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232913.329505:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232913.329517:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895096256/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:2.0:1761232913.329521:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761232913.329531:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232913.329535:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232913.329537:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232913.329543:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5474 00000100:00100000:2.0:1761232913.329544:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232913/real 1761232913] req@00000000c6e06344 x1846780895096128/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232938 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232913.329548:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232913.329555:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232913.329561:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232913.329561:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895096128/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232938 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232913.329573:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232913.329576:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232913.329578:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895096320/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232913.329582:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895096128:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761232913.329595:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232913.329599:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232913.329601:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232913.329606:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5474 00000100:00100000:2.0:1761232913.329609:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895096192:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:0.0:1761232913.329610:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232913.329619:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232913.329625:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1761232913.329631:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232913.329640:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895096384/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232913.329642:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232913/real 1761232913] req@000000005593c994 x1846780895096192/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232938 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232913.329676:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1761232913.329676:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895096192/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232938 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232913.329686:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232913.329689:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232913.329690:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232913.329695:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761232913.329698:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5474 00000100:00100000:2.0:1761232913.329701:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895096192:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:0.0:1761232913.329704:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1761232913.329708:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895096256:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00080000:0.0:1761232913.329712:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232913.329720:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1761232913.329726:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232913.329737:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232913/real 1761232913] req@000000007bd1ec64 x1846780895096256/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232938 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232913.329738:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895096448/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232913.329771:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1761232913.329773:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895096256/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232938 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232913.329778:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232913.329781:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232913.329783:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232913.329787:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761232913.329789:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5474 00000100:00100000:2.0:1761232913.329792:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895096256:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00080000:0.0:1761232913.329794:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1761232913.329800:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895096320:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00080000:0.0:1761232913.329803:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232913.329811:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1761232913.329824:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232913.329828:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895096512/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232913.329837:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232913/real 1761232913] req@000000001db96463 x1846780895096320/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232938 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232913.329851:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895096320/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232938 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232913.329860:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232913.329862:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232913.329866:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895096320:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232913.329889:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895096384:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232913.329913:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232913.329922:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232913/real 1761232913] req@00000000efa29eb9 x1846780895096384/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232938 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232913.329933:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895096384/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232938 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232913.329942:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232913.329944:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232913.329947:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895096384:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232913.329953:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895096448:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232913.329971:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232913.329979:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232913/real 1761232913] req@000000006d61dd06 x1846780895096448/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232938 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232913.329989:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895096448/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232938 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232913.329998:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232913.330000:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232913.330003:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895096448:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232913.330009:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895096512:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232913.330027:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232913.330035:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232913/real 1761232913] req@00000000ef655201 x1846780895096512/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232938 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232913.330045:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895096512/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232938 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232913.330053:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232913.330055:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232913.330058:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895096512:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761232914.097441:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232918.193420:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1761232918.449607:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895096576:192.168.201.157@tcp:250:kworker/u8:3.0 00000800:00000100:2.0:1761232918.449674:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232918.449697:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232918/real 1761232918] req@000000003b40830d x1846780895096576/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232943 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232918.449732:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895096576/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232943 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232918.449752:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232918.449756:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232918.449763:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895096576:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761232918.449772:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895096640:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761232918.449795:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232918.449803:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232918/real 1761232918] req@000000009706a782 x1846780895096640/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232943 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232918.449830:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895096640/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232943 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232918.449837:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232918.449838:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232918.449840:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895096640:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232918.449862:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895096704:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232918.449895:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232918.449901:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232918/real 1761232918] req@000000007aee79ba x1846780895096704/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232943 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232918.449909:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895096704/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232943 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232918.449915:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232918.449917:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232918.449919:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895096704:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232918.449924:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895096768:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232918.449936:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232918.449942:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232918/real 1761232918] req@000000008e58c4b8 x1846780895096768/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232943 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232918.449951:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895096768/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232943 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232918.449957:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232918.449958:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232918.449960:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895096768:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232918.449964:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895096832:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761232918.449977:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232918.449982:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232918/real 1761232918] req@0000000080131ea7 x1846780895096832/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232943 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232918.449990:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895096832/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232943 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232918.449995:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232918.449997:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232918.449999:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895096832:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232918.450003:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895096896:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232918.450015:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232918.450020:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232918/real 1761232918] req@000000007adf99ab x1846780895096896/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232943 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232918.450028:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895096896/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232943 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232918.450033:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232918.450036:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232918.450038:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895096896:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232918.450042:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895096960:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232918.450090:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232918.450113:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232918/real 1761232918] req@00000000e99e5a12 x1846780895096960/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232943 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232918.450124:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895096960/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232943 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232918.450131:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232918.450133:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232918.450136:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895096960:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232922.289460:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761232923.569361:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232923.569370:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232923.569373:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232923.569384:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5484 00000100:00080000:0.0:1761232923.569387:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232923.569395:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232923.569429:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895097024/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232923.569453:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232923.569456:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232923.569458:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232923.569464:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5484 00000100:00080000:0.0:1761232923.569465:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232923.569468:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232923.569475:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232923.569482:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895097088/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232923.569489:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232923.569492:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232923.569493:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232923.569496:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5484 00000100:00080000:0.0:1761232923.569497:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232923.569499:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232923.569502:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232923.569507:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895097152/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232923.569511:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895097024:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761232923.569515:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232923.569518:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232923.569518:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232923.569521:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5484 00000100:00080000:0.0:1761232923.569522:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232923.569525:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232923.569528:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232923.569537:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895097216/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232923.569542:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232923.569545:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232923.569546:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232923.569548:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5484 00000100:00080000:0.0:1761232923.569549:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232923.569552:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232923.569555:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232923.569560:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895097280/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232923.569565:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232923.569566:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232923.569567:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232923.569569:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5484 00000100:00080000:0.0:1761232923.569570:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232923.569574:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:2.0:1761232923.569577:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:0.0:1761232923.569579:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232923.569586:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895097344/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232923.569591:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232923.569593:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232923.569594:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232923.569597:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5484 00000100:00080000:0.0:1761232923.569599:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1761232923.569600:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232923/real 1761232923] req@000000004130b6bd x1846780895097024/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232948 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761232923.569604:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232923.569609:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232923.569618:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895097408/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232923.569621:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895097024/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232948 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232923.569637:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232923.569640:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232923.569643:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895097024:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761232923.569650:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895097088:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761232923.569668:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232923.569672:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232923/real 1761232923] req@000000001f9ec359 x1846780895097088/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232948 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232923.569680:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895097088/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232948 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232923.569684:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232923.569686:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232923.569688:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895097088:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232923.569708:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895097152:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232923.569719:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232923.569723:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232923/real 1761232923] req@000000009d1c7f73 x1846780895097152/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232948 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232923.569729:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895097152/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232948 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232923.569734:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232923.569736:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232923.569739:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895097152:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232923.569744:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895097216:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232923.569760:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232923.569767:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232923/real 1761232923] req@00000000d67bb589 x1846780895097216/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232948 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232923.569777:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895097216/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232948 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232923.569784:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232923.569786:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232923.569789:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895097216:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232923.569795:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895097280:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761232923.569814:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232923.569821:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232923/real 1761232923] req@00000000a4a2c44a x1846780895097280/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232948 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232923.569830:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895097280/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232948 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232923.569836:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232923.569838:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232923.569841:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895097280:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232923.569846:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895097344:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232923.569863:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232923.569871:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232923/real 1761232923] req@000000004a74da19 x1846780895097344/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232948 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232923.569880:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895097344/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232948 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232923.569887:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232923.569890:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232923.569892:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895097344:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232923.569899:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895097408:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232923.569917:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232923.569924:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232923/real 1761232923] req@000000002d1b8ef7 x1846780895097408/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232948 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232923.569931:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895097408/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232948 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232923.569936:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232923.569937:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232923.569939:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895097408:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232926.385493:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761232928.689316:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232928.689328:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232928.689332:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232928.689345:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5489 00000100:00080000:2.0:1761232928.689349:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232928.689356:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232928.689390:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895097472/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232928.689425:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232928.689428:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232928.689429:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232928.689431:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5489 00000100:00080000:2.0:1761232928.689433:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232928.689436:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232928.689446:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232928.689454:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895097472:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761232928.689457:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895097536/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232928.689467:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232928.689469:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232928.689470:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232928.689473:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5489 00000100:00080000:2.0:1761232928.689474:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232928.689476:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232928.689479:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232928.689486:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895097600/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232928.689491:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232928.689493:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232928.689494:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232928.689496:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5489 00000100:00080000:2.0:1761232928.689498:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232928.689499:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232928.689502:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232928.689507:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895097664/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232928.689511:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232928.689513:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232928.689514:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232928.689517:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5489 00000100:00080000:2.0:1761232928.689518:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232928.689521:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232928.689524:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232928.689532:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895097728/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761232928.689533:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761232928.689540:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232928.689542:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232928.689544:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232928.689547:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5489 00000100:00080000:2.0:1761232928.689549:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232928.689552:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232928.689554:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232928.689561:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895097792/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232928.689564:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232928/real 1761232928] req@000000004f29ca26 x1846780895097472/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232953 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232928.689572:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232928.689578:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232928.689580:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232928.689586:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5489 00000100:00080000:2.0:1761232928.689587:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232928.689590:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232928.689593:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232928.689595:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895097472/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232953 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232928.689602:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895097856/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761232928.689607:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232928.689611:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232928.689618:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895097472:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761232928.689642:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895097536:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761232928.689660:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232928.689666:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232928/real 1761232928] req@0000000033765045 x1846780895097536/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232953 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232928.689673:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895097536/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232953 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232928.689677:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232928.689679:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232928.689681:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895097536:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232928.689684:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895097600:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232928.689705:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232928.689710:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232928/real 1761232928] req@000000009d315fa5 x1846780895097600/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232953 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232928.689717:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895097600/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232953 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232928.689722:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232928.689723:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232928.689725:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895097600:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232928.689728:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895097664:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232928.689739:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232928.689747:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232928/real 1761232928] req@0000000040d5f361 x1846780895097664/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232953 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232928.689753:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895097664/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232953 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232928.689758:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232928.689759:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232928.689761:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895097664:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232928.689764:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895097728:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761232928.689774:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232928.689777:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232928/real 1761232928] req@000000007dcf84ae x1846780895097728/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232953 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232928.689783:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895097728/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232953 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232928.689787:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232928.689789:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232928.689791:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895097728:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232928.689794:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895097792:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232928.689804:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232928.689807:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232928/real 1761232928] req@000000007091f547 x1846780895097792/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232953 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232928.689813:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895097792/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232953 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232928.689817:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232928.689819:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232928.689820:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895097792:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232928.689823:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895097856:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232928.689832:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232928.689835:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232928/real 1761232928] req@0000000064a2cea0 x1846780895097856/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232953 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232928.689841:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895097856/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232953 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232928.689845:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232928.689846:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232928.689848:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895097856:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232930.481415:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761232933.809324:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232933.809336:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232933.809341:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232933.809353:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5494 00000100:00080000:2.0:1761232933.809357:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232933.809364:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232933.809400:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895097920/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232933.809426:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232933.809429:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232933.809430:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232933.809433:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5494 00000100:00080000:2.0:1761232933.809435:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232933.809437:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232933.809445:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232933.809454:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895097984/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232933.809460:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232933.809462:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232933.809463:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232933.809465:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5494 00000100:00080000:2.0:1761232933.809466:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232933.809468:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232933.809472:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232933.809476:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895098048/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232933.809481:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232933.809483:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232933.809485:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232933.809487:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5494 00000100:00080000:2.0:1761232933.809488:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232933.809491:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232933.809493:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232933.809498:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895098112/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232933.809503:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232933.809506:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232933.809506:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232933.809510:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5494 00000100:00100000:1.0:1761232933.809512:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895097920:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761232933.809513:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232933.809518:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232933.809521:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232933.809528:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895098176/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232933.809534:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232933.809536:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232933.809537:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232933.809540:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5494 00000100:00080000:2.0:1761232933.809542:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232933.809544:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232933.809547:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232933.809553:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895098240/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232933.809557:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232933.809559:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232933.809559:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232933.809561:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5494 00000100:00080000:2.0:1761232933.809563:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232933.809565:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232933.809567:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232933.809574:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895098304/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761232933.809578:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232933.809599:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232933/real 1761232933] req@0000000064a2cea0 x1846780895097920/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232958 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232933.809612:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895097920/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232958 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232933.809619:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232933.809621:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232933.809623:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895097920:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761232933.809628:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895097984:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761232933.809640:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232933.809645:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232933/real 1761232933] req@000000007091f547 x1846780895097984/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232958 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232933.809651:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895097984/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232958 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232933.809655:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232933.809657:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232933.809658:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895097984:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232933.809661:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895098048:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232933.809671:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232933.809674:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232933/real 1761232933] req@000000007dcf84ae x1846780895098048/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232958 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232933.809680:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895098048/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232958 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232933.809685:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232933.809687:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232933.809688:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895098048:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232933.809709:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895098112:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232933.809718:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232933.809722:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232933/real 1761232933] req@0000000040d5f361 x1846780895098112/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232958 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232933.809728:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895098112/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232958 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232933.809732:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232933.809733:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232933.809735:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895098112:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232933.809738:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895098176:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761232933.809747:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232933.809751:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232933/real 1761232933] req@000000009d315fa5 x1846780895098176/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232958 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232933.809757:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895098176/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232958 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232933.809761:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232933.809762:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232933.809764:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895098176:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232933.809768:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895098240:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232933.809776:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232933.809780:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232933/real 1761232933] req@0000000033765045 x1846780895098240/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232958 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232933.809785:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895098240/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232958 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232933.809790:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232933.809791:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232933.809793:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895098240:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232933.809795:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895098304:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232933.809804:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232933.809807:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232933/real 1761232933] req@000000004f29ca26 x1846780895098304/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232958 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232933.809813:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895098304/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232958 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232933.809817:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232933.809818:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232933.809820:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895098304:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232934.577376:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232938.673498:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761232938.929355:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232938.929364:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232938.929371:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232938.929384:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5499 00000100:00080000:2.0:1761232938.929388:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232938.929396:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232938.929435:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895098368/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232938.929466:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232938.929470:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232938.929472:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232938.929476:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5499 00000100:00080000:2.0:1761232938.929479:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232938.929485:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232938.929497:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232938.929517:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895098368:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761232938.929517:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895098432/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232938.929542:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232938.929545:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232938.929546:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232938.929551:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5499 00000100:00080000:2.0:1761232938.929553:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232938.929555:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232938.929559:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232938.929569:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895098496/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232938.929575:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232938.929577:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232938.929579:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232938.929581:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5499 00000100:00080000:2.0:1761232938.929583:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232938.929586:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232938.929590:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232938.929595:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895098560/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232938.929601:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232938.929603:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232938.929604:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232938.929607:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5499 00000100:00080000:2.0:1761232938.929609:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232938.929611:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232938.929613:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761232938.929616:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232938.929626:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895098624/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232938.929635:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232938.929637:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232938.929638:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232938.929641:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5499 00000100:00080000:2.0:1761232938.929644:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761232938.929654:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232938/real 1761232938] req@00000000fd3935a6 x1846780895098368/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232963 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232938.929655:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232938.929666:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232938.929682:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895098688/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232938.929688:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895098368/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232963 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232938.929710:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232938.929725:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232938.929728:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761232938.929735:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232938.929735:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5499 00000100:00080000:2.0:1761232938.929746:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761232938.929747:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232938.929756:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895098368:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761232938.929759:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232938.929801:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232938.929804:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895098432:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232938.929826:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895098752/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761232938.929832:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232938.929841:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232938/real 1761232938] req@00000000135e9a6e x1846780895098432/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232963 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232938.929853:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895098432/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232963 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232938.929859:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232938.929861:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232938.929863:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895098432:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232938.929868:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895098496:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232938.929884:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232938.929888:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232938/real 1761232938] req@000000006ff1fdbc x1846780895098496/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232963 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232938.929895:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895098496/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232963 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232938.929904:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232938.929905:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232938.929907:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895098496:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232938.929910:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895098560:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232938.929920:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232938.929924:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232938/real 1761232938] req@0000000088939481 x1846780895098560/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232963 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232938.929930:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895098560/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232963 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232938.929934:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232938.929936:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232938.929937:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895098560:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232938.929940:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895098624:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761232938.929950:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232938.929954:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232938/real 1761232938] req@000000006d8c3eb0 x1846780895098624/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232963 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232938.929960:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895098624/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232963 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232938.929964:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232938.929966:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232938.929968:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895098624:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232938.929972:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895098688:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232938.929982:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232938.929985:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232938/real 1761232938] req@000000006c1041ba x1846780895098688/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232963 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232938.929991:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895098688/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232963 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232938.929996:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232938.929997:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232938.929999:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895098688:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232938.930024:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895098752:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232938.930033:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232938.930037:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232938/real 1761232938] req@00000000fd4aa3a2 x1846780895098752/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232963 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232938.930044:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895098752/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232963 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232938.930049:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232938.930050:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232938.930051:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895098752:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232942.769433:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1761232944.049600:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895098816:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761232944.049735:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895098880:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761232944.049764:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895098944:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232944.049789:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895099008:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232944.049806:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895099072:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761232944.049822:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895099136:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232944.049834:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895099200:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761232946.865383:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1761232947.121624:0:2210:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:1.0:1761232947.121691:0:2210:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1761232947.121708:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761232947.121751:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761232947.121760:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761232947.121767:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761232947.121775:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761232947.121783:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761232947.121793:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:2.0:1761232947.121816:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232944/real 1761232947] req@00000000e99e5a12 x1846780895099008/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232969 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232947.121844:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895099008/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232969 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232947.121854:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232947.121858:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232947.121862:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895099008:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232947.121869:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232944/real 1761232947] req@000000007adf99ab x1846780895099072/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232969 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232947.121881:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895099072/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232969 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232947.121888:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232947.121890:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232947.121893:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895099072:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232947.121898:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232944/real 1761232947] req@0000000080131ea7 x1846780895099136/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232969 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232947.121906:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895099136/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232969 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232947.121913:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232947.121915:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232947.121918:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895099136:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232947.121923:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232944/real 1761232947] req@000000008e58c4b8 x1846780895099200/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232969 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232947.121933:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895099200/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232969 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232947.121939:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232947.121941:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232947.121944:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895099200:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232947.121948:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232944/real 1761232947] req@00000000b6fdd7f9 x1846780895098816/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232969 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232947.121959:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895098816/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232969 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232947.121966:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232947.121968:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232947.121970:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895098816:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761232947.121975:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232944/real 1761232947] req@0000000090702bc4 x1846780895098880/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232969 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232947.121984:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895098880/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232969 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232947.121996:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232947.121998:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232947.122001:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895098880:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232947.122005:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232944/real 1761232947] req@0000000070fa3bd6 x1846780895098944/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232969 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232947.122015:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895098944/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232969 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232947.122020:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232947.122022:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232947.122025:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895098944:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232949.169492:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895099264:192.168.201.157@tcp:250:kworker/u8:3.0 00000800:00000100:2.0:1761232949.169572:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232949.169596:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232949/real 1761232949] req@000000007aee79ba x1846780895099264/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232974 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232949.169653:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895099264/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232974 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232949.169671:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232949.169677:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232949.169684:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895099264:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761232949.169695:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895099328:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761232949.169728:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232949.169743:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232949/real 1761232949] req@000000009706a782 x1846780895099328/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232974 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232949.169760:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895099328/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232974 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232949.169772:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232949.169774:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232949.169778:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895099328:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232949.169810:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895099392:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232949.169831:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232949.169838:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232949/real 1761232949] req@000000003b40830d x1846780895099392/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232974 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232949.169849:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895099392/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232974 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232949.169857:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232949.169859:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232949.169862:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895099392:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232949.169868:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895099456:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232949.169885:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232949.169891:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232949/real 1761232949] req@00000000c9993cd4 x1846780895099456/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232974 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232949.169901:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895099456/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232974 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232949.169913:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232949.169915:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232949.169918:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895099456:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232949.169923:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895099520:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761232949.169943:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232949.169951:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232949/real 1761232949] req@000000002625ec0b x1846780895099520/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232974 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232949.169961:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895099520/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232974 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232949.169969:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232949.169972:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232949.169975:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895099520:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232949.169981:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895099584:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232949.169998:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232949.170005:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232949/real 1761232949] req@00000000bbf6af7b x1846780895099584/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232974 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232949.170015:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895099584/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232974 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232949.170022:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232949.170024:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232949.170027:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895099584:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232949.170033:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895099648:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232949.170051:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232949.170058:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232949/real 1761232949] req@00000000829a5595 x1846780895099648/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232974 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232949.170069:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895099648/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232974 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232949.170076:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232949.170078:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232949.170081:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895099648:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232950.961418:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761232954.289343:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232954.289355:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232954.289359:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232954.289371:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5515 00000100:00080000:2.0:1761232954.289375:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232954.289382:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232954.289427:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895099712/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232954.289453:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232954.289456:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232954.289457:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232954.289460:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5515 00000100:00080000:2.0:1761232954.289461:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232954.289463:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232954.289474:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232954.289482:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895099776/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232954.289488:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895099712:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761232954.289492:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232954.289496:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232954.289497:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232954.289501:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5515 00000100:00080000:2.0:1761232954.289502:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232954.289504:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232954.289508:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232954.289517:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895099840/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232954.289521:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232954.289524:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232954.289525:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232954.289544:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5515 00000100:00080000:2.0:1761232954.289546:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232954.289550:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232954.289553:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232954.289561:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895099904/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232954.289566:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232954.289569:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232954.289570:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232954.289573:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5515 00000100:00080000:2.0:1761232954.289575:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232954.289578:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232954.289581:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761232954.289585:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232954.289589:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895099968/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232954.289595:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232954.289598:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232954.289599:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232954.289603:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5515 00000100:00080000:2.0:1761232954.289604:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232954.289608:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232954.289610:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232954.289615:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232954/real 1761232954] req@00000000f4e415d3 x1846780895099712/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232979 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232954.289618:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895100032/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232954.289635:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232954.289638:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232954.289639:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232954.289642:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5515 00000100:00100000:1.0:1761232954.289645:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895099712/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232979 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232954.289646:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232954.289653:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232954.289658:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232954.289667:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895100096/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761232954.289670:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232954.289676:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232954.289682:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895099712:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761232954.289708:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895099776:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761232954.289724:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232954.289729:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232954/real 1761232954] req@0000000099b7e5d9 x1846780895099776/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232979 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232954.289736:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895099776/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232979 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232954.289741:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232954.289743:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232954.289745:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895099776:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232954.289748:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895099840:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232954.289759:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232954.289762:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232954/real 1761232954] req@00000000c10cebe6 x1846780895099840/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232979 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232954.289769:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895099840/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232979 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232954.289773:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232954.289775:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232954.289776:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895099840:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232954.289780:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895099904:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232954.289790:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232954.289794:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232954/real 1761232954] req@00000000a1e8c6bb x1846780895099904/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232979 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232954.289800:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895099904/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232979 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232954.289805:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232954.289807:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232954.289808:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895099904:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232954.289811:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895099968:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761232954.289822:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232954.289826:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232954/real 1761232954] req@000000008ebccab5 x1846780895099968/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232979 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232954.289831:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895099968/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232979 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232954.289836:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232954.289837:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232954.289839:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895099968:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232954.289842:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895100032:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232954.289851:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232954.289855:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232954/real 1761232954] req@000000004ae5bc9b x1846780895100032/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232979 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232954.289862:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895100032/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232979 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232954.289866:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232954.289867:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232954.289869:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895100032:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232954.289872:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895100096:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232954.289881:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232954.289885:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232954/real 1761232954] req@00000000ac7d8d5b x1846780895100096/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232979 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232954.289890:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895100096/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232979 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232954.289894:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232954.289895:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232954.289897:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895100096:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232955.057373:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232959.153417:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761232959.409326:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232959.409340:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232959.409345:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232959.409360:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5520 00000100:00080000:2.0:1761232959.409365:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232959.409373:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232959.409425:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895100160/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232959.409492:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232959.409497:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232959.409499:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232959.409504:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5520 00000100:00080000:2.0:1761232959.409507:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232959.409513:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232959.409526:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232959.409535:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895100160:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761232959.409545:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895100224/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232959.409559:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232959.409563:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232959.409565:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232959.409570:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5520 00000100:00080000:2.0:1761232959.409573:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232959.409577:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232959.409587:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232959.409598:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895100288/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761232959.409610:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761232959.409612:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232959.409617:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232959.409619:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232959.409626:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5520 00000100:00080000:2.0:1761232959.409629:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232959.409634:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761232959.409638:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232959/real 1761232959] req@00000000ac7d8d5b x1846780895100160/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232984 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:2.0:1761232959.409643:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232959.409661:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895100160/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232984 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232959.409668:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895100352/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761232959.409701:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232959.409703:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761232959.409706:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761232959.409712:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1761232959.409713:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895100160:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761232959.409715:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232959.409725:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5520 00000100:00080000:2.0:1761232959.409728:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232959.409733:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232959.409739:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232959.409742:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895100224:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232959.409752:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895100416/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761232959.409764:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761232959.409770:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761232959.409776:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232959/real 1761232959] req@000000004ae5bc9b x1846780895100224/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232984 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232959.409776:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232959.409782:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232959.409796:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5520 00000100:00100000:1.0:1761232959.409800:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895100224/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232984 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232959.409802:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761232959.409812:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232959.409812:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:1.0:1761232959.409823:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000008:00100000:2.0:1761232959.409827:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232959.409831:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895100224:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232959.409839:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895100288:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232959.409848:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895100480/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761232959.409859:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232959.409866:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232959/real 1761232959] req@000000008ebccab5 x1846780895100288/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232984 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232959.409869:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232959.409882:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232959.409886:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761232959.409892:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895100288/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232984 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232959.409899:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5520 00000100:00080000:2.0:1761232959.409907:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761232959.409909:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232959.409914:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761232959.409919:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761232959.409923:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895100288:192.168.201.157@tcp:8:kworker/u8:3.0 00000008:00100000:2.0:1761232959.409928:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232959.409933:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895100352:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232959.409952:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895100544/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761232959.409958:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232959.409968:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232959/real 1761232959] req@00000000a1e8c6bb x1846780895100352/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232984 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232959.409983:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895100352/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232984 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232959.409987:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232959.409988:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232959.409991:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895100352:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232959.410009:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895100416:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761232959.410019:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232959.410023:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232959/real 1761232959] req@00000000c10cebe6 x1846780895100416/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232984 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232959.410029:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895100416/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232984 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232959.410033:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232959.410035:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232959.410036:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895100416:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232959.410040:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895100480:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232959.410049:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232959.410052:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232959/real 1761232959] req@0000000099b7e5d9 x1846780895100480/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232984 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232959.410059:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895100480/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232984 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232959.410063:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232959.410064:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232959.410066:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895100480:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232959.410069:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895100544:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232959.410078:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232959.410081:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232959/real 1761232959] req@00000000f4e415d3 x1846780895100544/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232984 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232959.410087:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895100544/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232984 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232959.410091:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232959.410092:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232959.410094:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895100544:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232960.177476:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232963.249444:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232964.273427:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761232964.529356:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232964.529365:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232964.529380:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232964.529390:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5525 00000100:00080000:0.0:1761232964.529394:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232964.529400:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232964.529432:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895100608/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232964.529453:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232964.529455:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232964.529457:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232964.529459:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5525 00000100:00080000:0.0:1761232964.529461:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232964.529463:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232964.529469:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232964.529477:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895100672/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232964.529483:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232964.529486:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232964.529488:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232964.529490:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5525 00000100:00080000:0.0:1761232964.529492:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232964.529496:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761232964.529498:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895100608:192.168.201.157@tcp:250:kworker/u8:2.0 00000008:00100000:0.0:1761232964.529500:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232964.529510:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895100736/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232964.529516:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232964.529518:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232964.529519:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232964.529521:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5525 00000100:00080000:0.0:1761232964.529522:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232964.529524:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232964.529526:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232964.529533:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895100800/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232964.529538:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232964.529540:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232964.529541:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232964.529543:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5525 00000100:00080000:0.0:1761232964.529544:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232964.529547:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232964.529550:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232964.529555:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895100864/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232964.529560:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232964.529563:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232964.529564:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232964.529566:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5525 00000100:00080000:0.0:1761232964.529567:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232964.529569:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232964.529572:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761232964.529573:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761232964.529580:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895100928/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232964.529586:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232964.529588:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232964.529589:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232964.529591:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5525 00000100:00080000:0.0:1761232964.529592:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232964.529594:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232964.529597:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232964.529602:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232964/real 1761232964] req@00000000f3ed2588 x1846780895100608/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232989 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761232964.529604:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895100992/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761232964.529637:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f3ed2588 x1846780895100608/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232989 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232964.529649:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232964.529652:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232964.529656:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895100608:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761232964.529664:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895100672:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761232964.529688:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232964.529696:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232964/real 1761232964] req@000000005d83493f x1846780895100672/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232989 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232964.529708:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005d83493f x1846780895100672/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232989 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232964.529716:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232964.529718:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232964.529721:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895100672:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761232964.529743:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895100736:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232964.529763:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232964.529770:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232964/real 1761232964] req@000000002d1b8ef7 x1846780895100736/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232989 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232964.529781:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895100736/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232989 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232964.529788:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232964.529791:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232964.529793:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895100736:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232964.529799:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895100800:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232964.529815:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232964.529822:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232964/real 1761232964] req@000000004a74da19 x1846780895100800/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232989 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232964.529832:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895100800/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232989 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232964.529839:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232964.529841:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232964.529844:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895100800:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232964.529849:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895100864:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761232964.529865:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232964.529871:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232964/real 1761232964] req@00000000a4a2c44a x1846780895100864/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232989 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232964.529881:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895100864/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232989 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232964.529888:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232964.529891:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232964.529894:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895100864:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761232964.529899:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895100928:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232964.529917:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232964.529923:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232964/real 1761232964] req@00000000d67bb589 x1846780895100928/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232989 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232964.529934:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895100928/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232989 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232964.529941:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232964.529943:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232964.529946:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895100928:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232964.529951:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895100992:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232964.529967:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232964.529974:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232964/real 1761232964] req@000000009d1c7f73 x1846780895100992/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232989 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232964.529983:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895100992/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232989 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232964.529991:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232964.529992:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232964.529995:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895100992:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761232967.345360:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232968.369404:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1761232969.649502:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895101056:192.168.201.157@tcp:250:kworker/u8:2.0 00000800:00000100:1.0:1761232969.649583:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232969.649612:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232969/real 1761232969] req@0000000057e55ffa x1846780895101056/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232994 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232969.649631:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895101056/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232994 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232969.649641:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232969.649646:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232969.649649:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895101056:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761232969.649656:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895101120:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761232969.649679:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232969.649688:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232969/real 1761232969] req@0000000089b8c4f1 x1846780895101120/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232994 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232969.649699:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895101120/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232994 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232969.649706:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232969.649708:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232969.649711:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895101120:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761232969.649735:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895101184:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232969.649755:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232969.649763:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232969/real 1761232969] req@000000002a0ba65f x1846780895101184/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232994 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232969.649772:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895101184/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232994 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232969.649778:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232969.649780:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232969.649783:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895101184:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232969.649789:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895101248:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232969.649806:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232969.649814:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232969/real 1761232969] req@00000000829a5595 x1846780895101248/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232994 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232969.649824:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895101248/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232994 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232969.649830:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232969.649832:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232969.649835:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895101248:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232969.649841:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895101312:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761232969.649858:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232969.649866:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232969/real 1761232969] req@00000000bbf6af7b x1846780895101312/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232994 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232969.649878:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895101312/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232994 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232969.649884:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232969.649886:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232969.649889:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895101312:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761232969.649894:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895101376:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232969.649911:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232969.649918:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232969/real 1761232969] req@000000002625ec0b x1846780895101376/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232994 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232969.649929:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895101376/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232994 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232969.649936:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232969.649938:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232969.649940:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895101376:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232969.649945:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895101440:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232969.649961:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232969.649968:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232969/real 1761232969] req@00000000c9993cd4 x1846780895101440/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232994 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232969.649977:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895101440/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232994 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232969.649984:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232969.649985:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232969.649988:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895101440:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761232971.441404:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232972.465420:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1761232974.769504:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895101504:192.168.201.157@tcp:250:kworker/u8:2.0 00000800:00000100:1.0:1761232974.769580:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232974.769609:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232974/real 1761232974] req@000000003b40830d x1846780895101504/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232999 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232974.769638:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895101504/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761232999 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232974.769655:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232974.769660:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232974.769669:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895101504:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761232974.769683:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895101568:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761232974.769714:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232974.769728:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232974/real 1761232974] req@000000009706a782 x1846780895101568/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232999 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232974.769742:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895101568/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232999 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232974.769750:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232974.769752:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232974.769756:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895101568:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761232974.769781:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895101632:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232974.769803:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232974.769811:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232974/real 1761232974] req@000000007aee79ba x1846780895101632/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232999 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232974.769822:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895101632/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232999 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232974.769829:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232974.769832:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232974.769834:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895101632:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232974.769840:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895101696:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232974.769857:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232974.769862:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232974/real 1761232974] req@0000000070fa3bd6 x1846780895101696/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232999 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232974.769869:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895101696/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232999 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232974.769873:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232974.769875:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232974.769876:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895101696:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232974.769879:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895101760:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761232974.769889:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232974.769893:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232974/real 1761232974] req@0000000090702bc4 x1846780895101760/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232999 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232974.769899:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895101760/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761232999 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232974.769904:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232974.769905:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232974.769906:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895101760:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761232974.769909:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895101824:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232974.769918:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232974.769922:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232974/real 1761232974] req@00000000b6fdd7f9 x1846780895101824/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232999 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232974.769928:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895101824/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232999 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232974.769935:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232974.769937:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232974.769940:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895101824:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761232974.769945:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895101888:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761232974.769964:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232974.769973:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232974/real 1761232974] req@000000008e58c4b8 x1846780895101888/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232999 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761232974.769984:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895101888/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761232999 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761232974.769992:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232974.769993:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232974.769997:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895101888:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761232975.537391:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232976.561476:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232979.633532:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1761232979.889510:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895101952:192.168.201.157@tcp:250:kworker/u8:2.0 00000800:00000100:2.0:1761232979.889621:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232979.889669:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232979/real 1761232979] req@0000000080131ea7 x1846780895101952/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233004 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232979.889749:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895101952/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233004 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232979.889781:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232979.889786:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232979.889792:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895101952:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761232979.889829:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895102016:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232979.889853:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232979.889861:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232979/real 1761232979] req@000000007adf99ab x1846780895102016/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233004 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232979.889871:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895102016/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233004 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232979.889878:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232979.889880:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232979.889882:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895102016:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232979.889888:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895102080:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232979.889905:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232979.889911:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232979/real 1761232979] req@00000000e99e5a12 x1846780895102080/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233004 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232979.889920:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895102080/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233004 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232979.889926:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232979.889927:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232979.889930:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895102080:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232979.889935:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895102144:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232979.889950:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232979.889955:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232979/real 1761232979] req@000000008e58c4b8 x1846780895102144/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233004 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232979.889964:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895102144/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233004 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232979.889970:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232979.889972:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232979.889975:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895102144:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232979.889980:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895102208:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232979.889997:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232979.890003:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232979/real 1761232979] req@00000000b6fdd7f9 x1846780895102208/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233004 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232979.890012:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895102208/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233004 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232979.890020:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232979.890022:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232979.890025:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895102208:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232979.890030:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895102272:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232979.890048:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232979.890056:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232979/real 1761232979] req@0000000090702bc4 x1846780895102272/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233004 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232979.890066:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895102272/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233004 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232979.890074:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232979.890077:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232979.890079:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895102272:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232979.890085:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895102336:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232979.890104:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232979.890111:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232979/real 1761232979] req@0000000070fa3bd6 x1846780895102336/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233004 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232979.890121:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895102336/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233004 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232979.890146:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232979.890148:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232979.890151:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895102336:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761232980.657413:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232983.729594:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232984.753409:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761232985.009403:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232985.009412:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761232985.009415:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232985.009424:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5545 00000100:00080000:0.0:1761232985.009427:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232985.009434:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761232985.009466:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895102400/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232985.009495:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232985.009497:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232985.009498:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232985.009501:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5545 00000100:00080000:0.0:1761232985.009502:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232985.009504:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232985.009513:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232985.009529:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895102464/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232985.009538:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232985.009541:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232985.009542:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232985.009546:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5545 00000100:00080000:0.0:1761232985.009548:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232985.009551:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232985.009557:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232985.009566:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895102400:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761232985.009575:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895102528/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232985.009593:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232985.009596:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232985.009598:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232985.009604:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5545 00000100:00080000:0.0:1761232985.009606:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232985.009611:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232985.009617:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232985.009628:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895102592/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232985.009635:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232985.009638:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761232985.009640:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232985.009643:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5545 00000100:00080000:0.0:1761232985.009645:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232985.009649:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232985.009654:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232985.009663:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895102656/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761232985.009671:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232985.009674:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761232985.009676:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761232985.009679:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5545 00000100:00080000:0.0:1761232985.009681:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1761232985.009687:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761232985.009693:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232985.009701:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761232985.009718:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895102720/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232985.009728:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232985/real 1761232985] req@000000001f9ec359 x1846780895102400/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233010 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232985.009822:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761232985.009827:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761232985.009828:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1761232985.009834:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895102400/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233010 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761232985.009835:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5545 00000100:00080000:0.0:1761232985.009847:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761232985.009864:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761232985.009875:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1761232985.009877:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232985.009884:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761232985.009896:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895102784/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232985.009897:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895102400:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761232985.009925:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895102464:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232985.009963:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232985.009972:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232985/real 1761232985] req@000000004130b6bd x1846780895102464/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233010 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232985.009987:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895102464/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233010 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232985.009996:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232985.009999:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232985.010003:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895102464:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232985.010040:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895102528:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232985.010117:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232985.010128:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232985/real 1761232985] req@00000000ef655201 x1846780895102528/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233010 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232985.010150:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895102528/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233010 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232985.010159:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232985.010162:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232985.010165:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895102528:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232985.010172:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895102592:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232985.010193:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232985.010231:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232985/real 1761232985] req@000000006d61dd06 x1846780895102592/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233010 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232985.010245:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895102592/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233010 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232985.010255:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232985.010258:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232985.010262:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895102592:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232985.010268:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895102656:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761232985.010292:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232985.010301:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232985/real 1761232985] req@00000000efa29eb9 x1846780895102656/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233010 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232985.010313:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895102656/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233010 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232985.010321:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232985.010324:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232985.010328:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895102656:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761232985.010334:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895102720:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232985.010352:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232985.010360:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232985/real 1761232985] req@000000001db96463 x1846780895102720/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233010 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232985.010371:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895102720/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233010 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232985.010380:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232985.010382:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232985.010385:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895102720:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761232985.010392:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895102784:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761232985.010411:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232985.010418:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232985/real 1761232985] req@000000007bd1ec64 x1846780895102784/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233010 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761232985.010430:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895102784/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233010 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761232985.010439:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232985.010442:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232985.010445:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895102784:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761232987.825519:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232988.849585:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761232990.129439:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761232990.129456:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761232990.129463:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761232990.129482:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5551 00000100:00080000:1.0:1761232990.129488:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761232990.129500:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761232990.129572:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004361f727 x1846780895102848/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761232990.129610:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761232990.129615:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761232990.129617:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761232990.129622:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5551 00000100:00080000:1.0:1761232990.129624:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761232990.129627:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761232990.129640:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232990.129652:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006a5e9865 x1846780895102912/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761232990.129661:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761232990.129664:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761232990.129666:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761232990.129670:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5551 00000100:00080000:1.0:1761232990.129673:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761232990.129676:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761232990.129681:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232990.129694:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac65ecb7 x1846780895102976/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761232990.129705:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761232990.129710:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761232990.129712:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761232990.129716:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5551 00000100:00080000:1.0:1761232990.129718:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761232990.129722:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761232990.129726:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232990.129742:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004361f727 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895102848:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761232990.129743:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df0b94a8 x1846780895103040/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761232990.129763:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761232990.129766:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761232990.129767:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761232990.129772:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5551 00000100:00080000:1.0:1761232990.129774:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761232990.129778:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761232990.129783:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232990.129795:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df5b2270 x1846780895103104/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761232990.129805:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761232990.129810:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761232990.129812:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761232990.129816:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5551 00000100:00080000:1.0:1761232990.129817:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761232990.129823:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761232990.129828:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232990.129840:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ffe86d04 x1846780895103168/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761232990.129849:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761232990.129852:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761232990.129854:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761232990.129865:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5551 00000800:00000100:2.0:1761232990.129868:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:1.0:1761232990.129871:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761232990.129878:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761232990.129883:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232990.129897:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d66930c2 x1846780895103232/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761232990.129912:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232990/real 1761232990] req@000000004361f727 x1846780895102848/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233015 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232990.129944:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004361f727 x1846780895102848/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233015 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232990.129958:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232990.129963:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232990.130009:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004361f727 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895102848:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761232990.130019:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a5e9865 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895102912:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761232990.130058:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232990.130068:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232990/real 1761232990] req@000000006a5e9865 x1846780895102912/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233015 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232990.130089:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006a5e9865 x1846780895102912/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233015 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232990.130097:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232990.130100:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232990.130104:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a5e9865 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895102912:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232990.130109:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac65ecb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895102976:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232990.130128:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232990.130135:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232990/real 1761232990] req@00000000ac65ecb7 x1846780895102976/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233015 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232990.130146:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac65ecb7 x1846780895102976/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233015 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232990.130155:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232990.130157:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232990.130159:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac65ecb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895102976:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232990.130282:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895103040:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232990.130309:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232990.130318:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232990/real 1761232990] req@00000000df0b94a8 x1846780895103040/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233015 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232990.130330:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df0b94a8 x1846780895103040/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233015 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232990.130338:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232990.130341:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232990.130344:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895103040:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232990.130351:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895103104:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761232990.130369:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232990.130376:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232990/real 1761232990] req@00000000df5b2270 x1846780895103104/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233015 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232990.130387:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df5b2270 x1846780895103104/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233015 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232990.130395:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232990.130397:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232990.130400:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895103104:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761232990.130406:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895103168:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232990.130421:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232990.130428:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232990/real 1761232990] req@00000000ffe86d04 x1846780895103168/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233015 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232990.130439:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ffe86d04 x1846780895103168/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233015 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232990.130447:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232990.130449:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232990.130452:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895103168:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761232990.130457:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895103232:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761232990.130477:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761232990.130484:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232990/real 1761232990] req@00000000d66930c2 x1846780895103232/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233015 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232990.130494:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d66930c2 x1846780895103232/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233015 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232990.130502:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761232990.130504:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761232990.130507:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895103232:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232991.921458:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232992.945539:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761232995.249478:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232995.249501:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761232995.249508:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232995.249528:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5556 00000100:00080000:2.0:1761232995.249535:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232995.249548:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761232995.249636:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895103296/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232995.249691:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232995.249699:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232995.249702:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232995.249707:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5556 00000100:00080000:2.0:1761232995.249710:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232995.249716:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232995.249732:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232995.249754:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895103360/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232995.249768:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232995.249774:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232995.249777:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761232995.249779:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895103296:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761232995.249786:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5556 00000100:00080000:2.0:1761232995.249793:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232995.249801:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232995.249807:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232995.249823:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895103424/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232995.249835:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232995.249839:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232995.249841:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232995.249846:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5556 00000100:00080000:2.0:1761232995.249848:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232995.249852:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232995.249857:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232995.249869:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895103488/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232995.249879:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232995.249882:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761232995.249917:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232995.249923:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5556 00000100:00080000:2.0:1761232995.249936:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232995.249941:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232995.249947:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761232995.249962:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895103552/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761232995.249975:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761232995.249983:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232995.249989:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761232995.249992:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761232995.249999:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5556 00000100:00080000:2.0:1761232995.250002:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232995.250007:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232995.250013:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761232995.250022:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232995/real 1761232995] req@00000000fd4aa3a2 x1846780895103296/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233020 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761232995.250029:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895103616/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761232995.250065:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761232995.250070:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761232995.250073:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761232995.250079:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895103296/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233020 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761232995.250084:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5556 00000100:00080000:2.0:1761232995.250091:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761232995.250100:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761232995.250109:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1761232995.250119:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1761232995.250127:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895103680/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761232995.250128:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232995.250143:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895103296:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761232995.250236:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895103360:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761232995.250277:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232995.250289:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232995/real 1761232995] req@000000006c1041ba x1846780895103360/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233020 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232995.250303:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895103360/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233020 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232995.250312:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232995.250316:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232995.250319:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895103360:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232995.250326:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895103424:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232995.250347:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232995.250356:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232995/real 1761232995] req@000000006d8c3eb0 x1846780895103424/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233020 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232995.250368:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895103424/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233020 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232995.250377:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232995.250380:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232995.250383:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895103424:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232995.250389:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895103488:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232995.250407:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232995.250415:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232995/real 1761232995] req@0000000088939481 x1846780895103488/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233020 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232995.250428:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895103488/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233020 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232995.250437:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232995.250439:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232995.250442:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895103488:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232995.250448:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895103552:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761232995.250466:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232995.250473:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232995/real 1761232995] req@000000006ff1fdbc x1846780895103552/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233020 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232995.250484:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895103552/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233020 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232995.250492:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232995.250495:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232995.250498:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895103552:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761232995.250504:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895103616:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232995.250521:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232995.250528:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232995/real 1761232995] req@00000000135e9a6e x1846780895103616/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233020 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232995.250539:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895103616/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233020 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232995.250547:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232995.250550:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232995.250553:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895103616:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761232995.250560:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895103680:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761232995.250577:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761232995.250585:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761232995/real 1761232995] req@00000000fd3935a6 x1846780895103680/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233020 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761232995.250596:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895103680/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233020 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761232995.250614:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761232995.250617:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761232995.250620:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895103680:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761232996.017535:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761232997.041496:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233000.113450:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233000.369399:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233000.369422:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233000.369429:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233000.369448:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5561 00000100:00080000:0.0:1761233000.369455:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233000.369467:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233000.369531:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895103744/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233000.369569:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233000.369573:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233000.369575:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233000.369580:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5561 00000100:00080000:0.0:1761233000.369582:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233000.369587:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233000.369603:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233000.369620:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895103808/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233000.369627:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895103744:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233000.369636:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233000.369642:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233000.369643:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233000.369649:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5561 00000100:00080000:0.0:1761233000.369651:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233000.369654:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233000.369659:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233000.369673:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895103872/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233000.369682:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233000.369685:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233000.369687:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233000.369691:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5561 00000100:00080000:0.0:1761233000.369693:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233000.369696:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233000.369701:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233000.369710:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895103936/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233000.369718:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233000.369721:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233000.369722:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233000.369727:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5561 00000100:00080000:0.0:1761233000.369729:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1761233000.369736:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233000.369737:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233000.369746:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233000.369759:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895104000/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233000.369768:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233000.369773:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1761233000.369774:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233000/real 1761233000] req@000000005593c994 x1846780895103744/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233025 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233000.369776:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233000.369788:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5561 00000100:00080000:0.0:1761233000.369792:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233000.369799:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233000.369826:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233000.369833:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895103744/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233025 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233000.369845:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895104064/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233000.369849:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233000.369857:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233000.369863:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233000.369869:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895103744:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233000.369871:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233000.369875:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233000.369883:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5561 00000100:00080000:0.0:1761233000.369886:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233000.369890:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233000.369895:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233000.369908:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895104128/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233000.369915:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895103808:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233000.369956:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233000.369969:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233000/real 1761233000] req@00000000c6e06344 x1846780895103808/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233025 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233000.370002:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895103808/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233025 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233000.370012:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233000.370016:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233000.370020:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895103808:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233000.370033:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895103872:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233000.370061:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233000.370069:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233000/real 1761233000] req@000000004862b808 x1846780895103872/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233025 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233000.370085:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895103872/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233025 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233000.370095:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233000.370098:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233000.370104:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895103872:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233000.370114:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895103936:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233000.370138:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233000.370146:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233000/real 1761233000] req@000000006f2efc8b x1846780895103936/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233025 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233000.370158:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895103936/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233025 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233000.370167:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233000.370169:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233000.370172:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895103936:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233000.370178:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895104000:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233000.370213:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233000.370219:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233000/real 1761233000] req@00000000eca9ddc2 x1846780895104000/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233025 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233000.370271:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895104000/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233025 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233000.370281:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233000.370284:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233000.370287:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895104000:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233000.370293:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895104064:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233000.370320:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233000.370333:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233000/real 1761233000] req@0000000034d49f59 x1846780895104064/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233025 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233000.370343:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895104064/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233025 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233000.370351:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233000.370354:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233000.370357:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895104064:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233000.370379:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895104128:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233000.370397:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233000.370403:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233000/real 1761233000] req@000000005beaae31 x1846780895104128/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233025 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233000.370414:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895104128/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233025 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233000.370421:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233000.370423:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233000.370425:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895104128:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233001.137487:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233004.209369:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233005.233467:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1761233005.489509:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895104192:192.168.201.157@tcp:250:kworker/u8:3.0 00000800:00000100:1.0:1761233005.489593:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233005.489622:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233005/real 1761233005] req@000000007aee79ba x1846780895104192/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233030 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233005.489648:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895104192/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233030 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233005.489669:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233005.489673:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233005.489678:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895104192:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233005.489703:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895104256:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233005.489718:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233005.489749:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233005/real 1761233005] req@000000009706a782 x1846780895104256/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233030 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233005.489758:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895104256/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233030 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233005.489763:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233005.489765:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233005.489767:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895104256:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233005.489771:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895104320:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233005.489785:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233005.489790:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233005/real 1761233005] req@000000003b40830d x1846780895104320/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233030 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233005.489796:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895104320/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233030 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233005.489801:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233005.489802:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233005.489804:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895104320:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233005.489808:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895104384:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233005.489818:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233005.489822:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233005/real 1761233005] req@00000000c9993cd4 x1846780895104384/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233030 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233005.489829:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895104384/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233030 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233005.489833:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233005.489834:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233005.489836:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895104384:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233005.489839:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895104448:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233005.489848:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233005.489852:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233005/real 1761233005] req@000000002625ec0b x1846780895104448/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233030 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233005.489857:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895104448/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233030 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233005.489861:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233005.489863:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233005.489865:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895104448:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233005.489868:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895104512:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233005.489877:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233005.489881:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233005/real 1761233005] req@00000000bbf6af7b x1846780895104512/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233030 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233005.489887:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895104512/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233030 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233005.489892:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233005.489894:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233005.489895:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895104512:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233005.489899:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895104576:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233005.489909:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233005.489912:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233005/real 1761233005] req@00000000829a5595 x1846780895104576/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233030 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233005.489918:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895104576/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233030 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233005.489923:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233005.489924:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233005.489926:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895104576:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233008.305517:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233009.329403:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1761233010.609526:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895104640:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233010.609639:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895104704:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233010.609682:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895104768:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233010.609712:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895104832:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233010.609786:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895104896:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233010.609874:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895104960:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233010.609933:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895105024:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233012.401548:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233013.425408:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1761233013.682068:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233010/real 1761233013] req@00000000c9993cd4 x1846780895105024/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233035 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233013.682104:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895105024/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233035 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233013.682121:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233013.682128:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233013.682133:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895105024:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233013.682143:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233010/real 1761233013] req@000000002625ec0b x1846780895104960/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233035 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233013.682160:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895104960/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233035 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233013.682171:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233013.682175:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233013.682180:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895104960:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233013.682188:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233010/real 1761233013] req@00000000bbf6af7b x1846780895104896/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233035 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233013.682234:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895104896/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233035 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233013.682246:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233013.682250:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233013.682281:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895104896:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233013.682292:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233010/real 1761233013] req@00000000829a5595 x1846780895104832/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233035 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233013.682312:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895104832/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233035 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233013.682324:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233013.682329:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233013.682335:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895104832:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233013.682344:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233010/real 1761233013] req@0000000089b8c4f1 x1846780895104704/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233035 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233013.682361:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895104704/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233035 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233013.682379:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233013.682382:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233013.682387:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895104704:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233013.682396:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233010/real 1761233013] req@0000000057e55ffa x1846780895104768/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233035 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233013.682411:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895104768/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233035 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233013.682422:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233013.682426:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233013.682430:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895104768:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233013.682439:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233010/real 1761233013] req@000000002a0ba65f x1846780895104640/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233035 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233013.682455:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895104640/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233035 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233013.682467:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233013.682472:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233013.682477:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895104640:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233015.729413:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895105088:192.168.201.157@tcp:250:kworker/u8:3.0 00000800:00000100:1.0:1761233015.729474:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233015.729495:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233015/real 1761233015] req@000000003b40830d x1846780895105088/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233040 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233015.729511:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895105088/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233040 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233015.729528:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233015.729532:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233015.729537:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895105088:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233015.729597:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895105152:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233015.729628:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233015.729641:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233015/real 1761233015] req@000000009706a782 x1846780895105152/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233040 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233015.729678:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895105152/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233040 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233015.729710:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233015.729714:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233015.729720:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895105152:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233015.729732:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895105216:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233015.729758:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233015.729769:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233015/real 1761233015] req@000000007aee79ba x1846780895105216/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233040 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233015.729780:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895105216/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233040 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233015.729787:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233015.729790:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233015.729792:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895105216:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233015.729798:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895105280:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233015.729815:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233015.729823:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233015/real 1761233015] req@0000000070fa3bd6 x1846780895105280/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233040 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233015.729833:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895105280/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233040 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233015.729840:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233015.729842:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233015.729844:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895105280:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233015.729863:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895105344:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233015.729886:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233015.729893:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233015/real 1761233015] req@0000000090702bc4 x1846780895105344/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233040 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233015.729904:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895105344/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233040 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233015.729911:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233015.729914:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233015.729917:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895105344:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233015.729922:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895105408:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233015.729940:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233015.729947:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233015/real 1761233015] req@00000000b6fdd7f9 x1846780895105408/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233040 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233015.729956:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895105408/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233040 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233015.729963:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233015.729965:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233015.729967:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895105408:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233015.729972:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895105472:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233015.729988:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233015.729995:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233015/real 1761233015] req@000000008e58c4b8 x1846780895105472/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233040 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233015.730006:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895105472/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233040 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233015.730012:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233015.730014:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233015.730017:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895105472:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233016.497363:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233017.521387:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233020.593462:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1761233020.849510:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895105536:192.168.201.157@tcp:250:kworker/u8:3.0 00000800:00000100:1.0:1761233020.849613:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233020.849655:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233020/real 1761233020] req@00000000e99e5a12 x1846780895105536/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233045 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233020.849740:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895105536/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233045 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233020.849765:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233020.849769:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233020.849778:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895105536:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233020.849809:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895105600:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233020.849835:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233020.849851:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233020/real 1761233020] req@000000007adf99ab x1846780895105600/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233045 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233020.849864:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895105600/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233045 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233020.849871:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233020.849873:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233020.849875:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895105600:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233020.849879:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895105664:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233020.849890:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233020.849894:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233020/real 1761233020] req@0000000080131ea7 x1846780895105664/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233045 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233020.849900:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895105664/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233045 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233020.849905:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233020.849906:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233020.849907:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895105664:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233020.849911:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895105728:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233020.849921:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233020.849925:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233020/real 1761233020] req@000000008e58c4b8 x1846780895105728/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233045 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233020.849931:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895105728/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233045 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233020.849936:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233020.849937:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233020.849938:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895105728:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233020.849943:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895105792:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233020.849954:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233020.849957:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233020/real 1761233020] req@00000000b6fdd7f9 x1846780895105792/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233045 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233020.849964:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895105792/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233045 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233020.849968:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233020.849970:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233020.849971:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895105792:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233020.849975:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895105856:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233020.849984:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233020.849988:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233020/real 1761233020] req@0000000090702bc4 x1846780895105856/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233045 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233020.849993:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895105856/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233045 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233020.849997:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233020.849998:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233020.850000:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895105856:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233020.850015:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895105920:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233020.850024:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233020.850028:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233020/real 1761233020] req@0000000070fa3bd6 x1846780895105920/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233045 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233020.850035:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895105920/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233045 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233020.850039:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233020.850040:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233020.850042:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895105920:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233021.617400:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233024.689437:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233025.713461:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1761233025.969440:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895105984:192.168.201.157@tcp:250:kworker/u8:3.0 00000800:00000100:1.0:1761233025.969533:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233025.969564:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233025/real 1761233025] req@000000007aee79ba x1846780895105984/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233050 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233025.969618:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895105984/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233050 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233025.969691:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233025.969695:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233025.969699:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895105984:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233025.969728:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895106048:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233025.969744:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233025.969748:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233025/real 1761233025] req@000000009706a782 x1846780895106048/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233050 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233025.969755:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895106048/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233050 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233025.969760:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233025.969762:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233025.969764:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895106048:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233025.969768:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895106112:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233025.969778:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233025.969782:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233025/real 1761233025] req@000000003b40830d x1846780895106112/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233050 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233025.969788:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895106112/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233050 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233025.969793:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233025.969794:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233025.969796:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895106112:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233025.969799:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895106176:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233025.969808:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233025.969811:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233025/real 1761233025] req@000000002a0ba65f x1846780895106176/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233050 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233025.969817:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895106176/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233050 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233025.969821:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233025.969823:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233025.969824:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895106176:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233025.969828:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895106240:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233025.969839:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233025.969842:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233025/real 1761233025] req@0000000057e55ffa x1846780895106240/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233050 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233025.969849:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895106240/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233050 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233025.969853:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233025.969854:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233025.969856:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895106240:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233025.969859:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895106304:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233025.969868:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233025.969872:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233025/real 1761233025] req@0000000089b8c4f1 x1846780895106304/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233050 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233025.969877:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895106304/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233050 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233025.969881:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233025.969883:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233025.969885:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895106304:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233025.969888:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895106368:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233025.969899:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233025.969903:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233025/real 1761233025] req@00000000829a5595 x1846780895106368/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233050 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233025.969909:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895106368/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233050 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233025.969914:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233025.969915:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233025.969917:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895106368:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233028.785332:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233029.809448:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1761233031.089424:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895106432:192.168.201.157@tcp:250:kworker/u8:3.0 00000800:00000100:1.0:1761233031.089501:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233031.089522:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233031/real 1761233031] req@00000000bbf6af7b x1846780895106432/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233056 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233031.089537:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895106432/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233056 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233031.089546:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233031.089551:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233031.089556:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895106432:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233031.089579:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895106496:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233031.089601:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233031.089612:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233031/real 1761233031] req@000000002625ec0b x1846780895106496/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233056 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233031.089628:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895106496/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233056 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233031.089653:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233031.089654:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233031.089657:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895106496:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233031.089661:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895106560:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233031.089674:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233031.089678:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233031/real 1761233031] req@00000000c9993cd4 x1846780895106560/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233056 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233031.089684:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895106560/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233056 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233031.089688:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233031.089689:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233031.089691:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895106560:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233031.089694:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895106624:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233031.089704:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233031.089708:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233031/real 1761233031] req@00000000829a5595 x1846780895106624/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233056 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233031.089714:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895106624/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233056 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233031.089718:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233031.089719:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233031.089721:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895106624:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233031.089724:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895106688:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233031.089734:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233031.089737:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233031/real 1761233031] req@0000000089b8c4f1 x1846780895106688/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233056 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233031.089744:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895106688/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233056 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233031.089748:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233031.089749:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233031.089750:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895106688:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233031.089761:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895106752:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233031.089771:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233031.089774:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233031/real 1761233031] req@0000000057e55ffa x1846780895106752/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233056 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233031.089780:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895106752/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233056 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233031.089784:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233031.089785:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233031.089787:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895106752:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233031.089790:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895106816:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233031.089798:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233031.089802:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233031/real 1761233031] req@000000002a0ba65f x1846780895106816/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233056 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233031.089808:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895106816/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233056 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233031.089812:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233031.089813:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233031.089815:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895106816:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233032.881378:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233033.905418:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233036.209387:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233036.209398:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233036.209401:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233036.209418:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5597 00000100:00080000:0.0:1761233036.209421:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233036.209428:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233036.209465:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000addc3db8 x1846780895106880/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233036.209489:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233036.209492:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233036.209493:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233036.209496:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5597 00000100:00080000:0.0:1761233036.209498:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233036.209500:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233036.209509:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233036.209519:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000078e7052d x1846780895106944/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233036.209527:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233036.209530:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233036.209531:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233036.209537:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5597 00000100:00100000:1.0:1761233036.209538:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895106880:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233036.209539:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233036.209545:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233036.209554:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233036.209564:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895107008/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233036.209570:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233036.209573:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233036.209574:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233036.209577:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5597 00000100:00080000:0.0:1761233036.209578:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233036.209581:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233036.209584:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233036.209591:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895107072/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233036.209597:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233036.209599:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233036.209600:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233036.209603:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5597 00000100:00080000:0.0:1761233036.209604:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233036.209607:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233036.209610:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233036.209620:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895107136/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761233036.209622:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233036.209630:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233036.209633:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233036.209634:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233036.209638:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5597 00000100:00080000:0.0:1761233036.209640:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233036.209643:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233036.209646:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233036.209653:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233036/real 1761233036] req@00000000addc3db8 x1846780895106880/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233061 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233036.209656:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895107200/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233036.209682:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233036.209689:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233036.209690:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233036.209694:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5597 00000100:00080000:0.0:1761233036.209695:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233036.209701:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761233036.209703:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000addc3db8 x1846780895106880/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233061 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000008:00100000:0.0:1761233036.209705:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233036.209715:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895107264/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233036.209726:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233036.209732:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233036.209738:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895106880:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761233036.209765:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895106944:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233036.209791:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233036.209801:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233036/real 1761233036] req@0000000078e7052d x1846780895106944/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233061 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233036.209815:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000078e7052d x1846780895106944/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233061 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233036.209833:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233036.209836:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233036.209839:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895106944:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233036.209846:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895107008:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233036.209867:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233036.209876:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233036/real 1761233036] req@000000005beaae31 x1846780895107008/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233061 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233036.209889:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895107008/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233061 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233036.209898:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233036.209900:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233036.209904:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895107008:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233036.209910:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895107072:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233036.209930:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233036.209938:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233036/real 1761233036] req@0000000034d49f59 x1846780895107072/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233061 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233036.209951:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895107072/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233061 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233036.209960:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233036.209962:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233036.209966:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895107072:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233036.209973:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895107136:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233036.209993:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233036.210001:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233036/real 1761233036] req@00000000eca9ddc2 x1846780895107136/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233061 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233036.210013:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895107136/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233061 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233036.210022:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233036.210025:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233036.210028:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895107136:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233036.210035:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895107200:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233036.210054:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233036.210062:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233036/real 1761233036] req@000000006f2efc8b x1846780895107200/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233061 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233036.210074:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895107200/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233061 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233036.210083:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233036.210086:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233036.210089:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895107200:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233036.210095:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895107264:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233036.210116:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233036.210126:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233036/real 1761233036] req@000000004862b808 x1846780895107264/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233061 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233036.210139:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895107264/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233061 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233036.210148:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233036.210150:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233036.210154:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895107264:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233036.977384:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233038.001368:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233041.073371:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233041.329301:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233041.329310:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233041.329313:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233041.329323:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5602 00000100:00080000:0.0:1761233041.329326:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233041.329333:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233041.329367:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895107328/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233041.329387:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233041.329391:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233041.329392:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233041.329394:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5602 00000100:00080000:0.0:1761233041.329396:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233041.329398:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233041.329405:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233041.329414:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895107392/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233041.329422:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895107328:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233041.329423:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233041.329430:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233041.329433:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233041.329436:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5602 00000100:00080000:0.0:1761233041.329438:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233041.329440:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233041.329443:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233041.329451:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895107456/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233041.329456:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233041.329458:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233041.329459:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233041.329461:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5602 00000100:00080000:0.0:1761233041.329462:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233041.329465:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233041.329467:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233041.329474:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895107520/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233041.329479:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233041.329482:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233041.329483:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233041.329484:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5602 00000100:00080000:0.0:1761233041.329485:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233041.329488:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233041.329490:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761233041.329494:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233041.329500:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895107584/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233041.329506:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233041.329509:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233041.329510:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233041.329512:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5602 00000100:00080000:0.0:1761233041.329513:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233041.329516:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233041.329520:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233041.329520:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233041/real 1761233041] req@00000000c6e06344 x1846780895107328/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233066 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233041.329530:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895107648/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233041.329541:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233041.329543:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233041.329544:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233041.329549:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5602 00000100:00100000:1.0:1761233041.329550:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895107328/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233066 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233041.329551:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233041.329556:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233041.329560:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233041.329569:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895107712/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233041.329570:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233041.329576:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233041.329584:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895107328:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761233041.329608:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895107392:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233041.329627:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233041.329634:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233041/real 1761233041] req@000000005593c994 x1846780895107392/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233066 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233041.329645:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895107392/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233066 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233041.329654:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233041.329656:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233041.329659:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895107392:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233041.329664:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895107456:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233041.329679:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233041.329684:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233041/real 1761233041] req@000000007bd1ec64 x1846780895107456/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233066 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233041.329694:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895107456/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233066 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233041.329702:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233041.329704:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233041.329706:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895107456:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233041.329711:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895107520:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233041.329725:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233041.329732:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233041/real 1761233041] req@000000001db96463 x1846780895107520/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233066 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233041.329741:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895107520/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233066 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233041.329748:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233041.329750:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233041.329753:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895107520:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233041.329759:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895107584:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233041.329775:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233041.329781:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233041/real 1761233041] req@00000000efa29eb9 x1846780895107584/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233066 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233041.329799:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895107584/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233066 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233041.329807:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233041.329809:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233041.329811:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895107584:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233041.329816:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895107648:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233041.329829:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233041.329835:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233041/real 1761233041] req@000000006d61dd06 x1846780895107648/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233066 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233041.329844:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895107648/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233066 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233041.329852:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233041.329854:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233041.329856:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895107648:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233041.329861:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895107712:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233041.329874:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233041.329880:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233041/real 1761233041] req@00000000ef655201 x1846780895107712/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233066 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233041.329890:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895107712/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233066 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233041.329897:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233041.329899:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233041.329901:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895107712:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233042.097389:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233045.169389:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233046.193469:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1761233046.449542:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895107776:192.168.201.157@tcp:250:kworker/u8:2.0 00000800:00000100:1.0:1761233046.449613:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233046.449636:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233046/real 1761233046] req@000000003b40830d x1846780895107776/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233071 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233046.449668:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895107776/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233071 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233046.449683:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233046.449688:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233046.449695:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895107776:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761233046.449719:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895107840:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233046.449743:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233046.449752:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233046/real 1761233046] req@000000009706a782 x1846780895107840/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233071 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233046.449766:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895107840/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233071 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233046.449784:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233046.449789:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233046.449794:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895107840:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233046.449802:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895107904:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233046.449821:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233046.449829:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233046/real 1761233046] req@000000007aee79ba x1846780895107904/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233071 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233046.449843:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895107904/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233071 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233046.449855:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233046.449857:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233046.449860:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895107904:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233046.449864:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895107968:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233046.449878:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233046.449882:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233046/real 1761233046] req@0000000070fa3bd6 x1846780895107968/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233071 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233046.449888:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895107968/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233071 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233046.449893:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233046.449895:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233046.449896:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895107968:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233046.449908:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895108032:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233046.449919:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233046.449923:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233046/real 1761233046] req@0000000090702bc4 x1846780895108032/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233071 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233046.449930:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895108032/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233071 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233046.449935:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233046.449936:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233046.449938:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895108032:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233046.449941:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895108096:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233046.449950:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233046.449954:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233046/real 1761233046] req@00000000b6fdd7f9 x1846780895108096/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233071 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233046.449960:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895108096/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233071 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233046.449965:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233046.449966:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233046.449968:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895108096:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233046.449971:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895108160:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233046.449980:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233046.449984:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233046/real 1761233046] req@000000008e58c4b8 x1846780895108160/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233071 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233046.449991:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895108160/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233071 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233046.449996:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233046.449997:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233046.449999:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895108160:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233049.265416:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233050.289475:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233051.569404:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233051.569413:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233051.569417:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233051.569426:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5612 00000100:00080000:0.0:1761233051.569430:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233051.569435:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233051.569466:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895108224/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233051.569487:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233051.569489:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233051.569490:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233051.569493:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5612 00000100:00080000:0.0:1761233051.569494:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233051.569497:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233051.569504:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233051.569512:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895108288/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233051.569519:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895108224:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233051.569521:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233051.569526:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233051.569527:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233051.569530:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5612 00000100:00080000:0.0:1761233051.569532:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233051.569534:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233051.569537:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233051.569545:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895108352/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233051.569550:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233051.569552:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233051.569553:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233051.569554:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5612 00000100:00080000:0.0:1761233051.569555:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233051.569557:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233051.569560:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233051.569566:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895108416/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233051.569570:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233051.569573:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233051.569574:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233051.569577:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5612 00000100:00080000:0.0:1761233051.569579:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233051.569583:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233051.569587:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761233051.569596:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233051.569600:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895108480/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233051.569611:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233051.569614:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233051.569616:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761233051.569619:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233051/real 1761233051] req@000000004130b6bd x1846780895108224/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233076 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233051.569624:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5612 00000100:00080000:0.0:1761233051.569629:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233051.569635:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233051.569640:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233051.569640:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895108224/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233076 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233051.569651:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1761233051.569654:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895108544/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233051.569656:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233051.569663:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895108224:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233051.569666:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233051.569669:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233051.569671:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233051.569674:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5612 00000100:00080000:0.0:1761233051.569676:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233051.569678:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233051.569680:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233051.569687:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895108288:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761233051.569689:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895108608/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761233051.569710:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233051.569718:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233051/real 1761233051] req@000000001f9ec359 x1846780895108288/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233076 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233051.569726:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895108288/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233076 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233051.569735:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233051.569736:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233051.569738:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895108288:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233051.569741:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895108352:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233051.569752:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233051.569755:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233051/real 1761233051] req@000000009d1c7f73 x1846780895108352/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233076 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233051.569761:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895108352/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233076 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233051.569766:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233051.569767:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233051.569769:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895108352:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233051.569772:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895108416:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233051.569781:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233051.569784:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233051/real 1761233051] req@00000000d67bb589 x1846780895108416/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233076 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233051.569790:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895108416/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233076 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233051.569794:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233051.569796:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233051.569797:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895108416:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233051.569802:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895108480:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233051.569811:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233051.569814:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233051/real 1761233051] req@00000000a4a2c44a x1846780895108480/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233076 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233051.569820:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895108480/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233076 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233051.569825:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233051.569826:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233051.569827:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895108480:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233051.569830:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895108544:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233051.569839:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233051.569842:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233051/real 1761233051] req@000000004a74da19 x1846780895108544/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233076 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233051.569847:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895108544/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233076 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233051.569852:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233051.569853:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233051.569855:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895108544:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233051.569868:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895108608:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233051.569877:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233051.569881:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233051/real 1761233051] req@000000002d1b8ef7 x1846780895108608/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233076 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233051.569887:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895108608/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233076 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233051.569891:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233051.569892:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233051.569894:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895108608:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233053.361418:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233054.385401:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1761233056.689488:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895108672:192.168.201.157@tcp:250:kworker/u8:3.0 00000800:00000100:1.0:1761233056.689558:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233056.689585:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233056/real 1761233056] req@0000000080131ea7 x1846780895108672/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233081 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233056.689627:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895108672/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233081 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233056.689648:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233056.689653:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233056.689661:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895108672:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233056.689690:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895108736:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233056.689711:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233056.689717:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233056/real 1761233056] req@000000007adf99ab x1846780895108736/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233081 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233056.689725:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895108736/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233081 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233056.689736:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233056.689737:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233056.689740:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895108736:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233056.689743:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895108800:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233056.689754:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233056.689758:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233056/real 1761233056] req@00000000e99e5a12 x1846780895108800/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233081 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233056.689766:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895108800/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233081 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233056.689770:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233056.689772:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233056.689773:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895108800:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233056.689777:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895108864:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233056.689787:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233056.689791:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233056/real 1761233056] req@000000008e58c4b8 x1846780895108864/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233081 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233056.689798:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895108864/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233081 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233056.689803:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233056.689804:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233056.689806:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895108864:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233056.689810:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895108928:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233056.689821:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233056.689826:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233056/real 1761233056] req@00000000b6fdd7f9 x1846780895108928/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233081 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233056.689833:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895108928/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233081 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233056.689838:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233056.689839:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233056.689841:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895108928:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233056.689844:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895108992:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233056.689854:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233056.689858:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233056/real 1761233056] req@0000000090702bc4 x1846780895108992/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233081 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233056.689864:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895108992/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233081 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233056.689876:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233056.689878:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233056.689879:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895108992:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233056.689894:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895109056:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233056.689905:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233056.689909:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233056/real 1761233056] req@0000000070fa3bd6 x1846780895109056/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233081 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233056.689944:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895109056/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233081 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233056.689950:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233056.689951:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233056.689954:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895109056:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233057.457346:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233058.481427:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233061.553389:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233061.809386:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233061.809395:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233061.809399:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233061.809408:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5622 00000100:00080000:0.0:1761233061.809411:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233061.809418:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233061.809450:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895109120/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233061.809472:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233061.809475:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233061.809476:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233061.809478:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5622 00000100:00080000:0.0:1761233061.809479:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233061.809482:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233061.809489:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233061.809497:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895109184/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233061.809504:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233061.809507:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1761233061.809508:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895109120:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233061.809509:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233061.809516:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5622 00000100:00080000:0.0:1761233061.809518:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233061.809520:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233061.809523:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233061.809532:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895109248/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233061.809537:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233061.809538:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233061.809540:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233061.809542:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5622 00000100:00080000:0.0:1761233061.809544:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233061.809546:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233061.809549:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233061.809554:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895109312/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233061.809558:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233061.809560:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233061.809562:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233061.809564:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5622 00000100:00080000:0.0:1761233061.809565:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233061.809567:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233061.809570:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761233061.809573:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233061.809581:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895109376/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233061.809587:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233061.809589:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233061.809590:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233061.809595:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5622 00000100:00100000:1.0:1761233061.809595:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233061/real 1761233061] req@000000005d83493f x1846780895109120/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233086 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233061.809599:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233061.809605:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233061.809610:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233061.809619:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895109440/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233061.809621:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005d83493f x1846780895109120/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233086 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233061.809643:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233061.809646:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233061.809647:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233061.809650:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233061.809653:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761233061.809653:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895109120:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233061.809661:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5622 00000100:00080000:0.0:1761233061.809662:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233061.809664:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233061.809667:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233061.809672:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895109504/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233061.809679:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895109184:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233061.809697:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233061.809703:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233061/real 1761233061] req@00000000f3ed2588 x1846780895109184/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233086 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233061.809710:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f3ed2588 x1846780895109184/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233086 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233061.809715:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233061.809717:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233061.809719:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895109184:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233061.809722:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895109248:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233061.809732:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233061.809735:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233061/real 1761233061] req@000000002d1b8ef7 x1846780895109248/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233086 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233061.809741:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895109248/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233086 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233061.809746:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233061.809747:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233061.809748:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895109248:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233061.809751:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895109312:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233061.809760:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233061.809764:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233061/real 1761233061] req@000000004a74da19 x1846780895109312/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233086 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233061.809770:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895109312/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233086 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233061.809774:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233061.809775:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233061.809777:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895109312:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233061.809781:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895109376:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233061.809790:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233061.809793:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233061/real 1761233061] req@00000000a4a2c44a x1846780895109376/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233086 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233061.809799:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895109376/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233086 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233061.809804:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233061.809805:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233061.809807:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895109376:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233061.809810:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895109440:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233061.809819:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233061.809822:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233061/real 1761233061] req@00000000d67bb589 x1846780895109440/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233086 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233061.809828:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895109440/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233086 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233061.809832:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233061.809833:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233061.809835:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895109440:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233061.809876:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895109504:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233061.809890:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233061.809894:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233061/real 1761233061] req@000000009d1c7f73 x1846780895109504/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233086 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233061.809901:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895109504/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233086 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233061.809906:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233061.809907:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233061.809909:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895109504:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233062.577550:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233065.649377:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233066.673445:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233066.929420:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233066.929430:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233066.929450:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233066.929459:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5627 00000100:00080000:1.0:1761233066.929462:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233066.929469:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233066.929504:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090ac513e x1846780895109568/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233066.929522:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233066.929525:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233066.929526:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233066.929529:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5627 00000100:00080000:1.0:1761233066.929530:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233066.929534:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233066.929541:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233066.929550:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895109568:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761233066.929553:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000053e52331 x1846780895109632/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233066.929562:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233066.929564:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233066.929565:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233066.929567:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5627 00000100:00080000:1.0:1761233066.929568:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233066.929570:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233066.929573:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233066.929580:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000880f16b4 x1846780895109696/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233066.929585:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233066.929587:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233066.929589:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233066.929591:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5627 00000100:00080000:1.0:1761233066.929592:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233066.929594:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233066.929596:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233066.929602:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cb5be845 x1846780895109760/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233066.929606:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233066.929609:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233066.929610:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233066.929612:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5627 00000100:00080000:1.0:1761233066.929614:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233066.929616:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233066.929619:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1761233066.929623:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233066.929629:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a95774fb x1846780895109824/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233066.929634:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233066.929637:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233066.929638:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233066.929641:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5627 00000100:00080000:1.0:1761233066.929642:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233066.929645:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233066.929647:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233066.929649:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233066/real 1761233066] req@0000000090ac513e x1846780895109568/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233091 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233066.929671:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000c736bb2 x1846780895109888/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233066.929683:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233066.929686:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233066.929687:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1761233066.929690:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090ac513e x1846780895109568/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233091 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233066.929694:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5627 00000100:00080000:1.0:1761233066.929699:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233066.929704:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:2.0:1761233066.929708:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1761233066.929709:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1761233066.929714:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233066.929720:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895109568:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761233066.929722:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000038209f5 x1846780895109952/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761233066.929749:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895109632:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233066.929806:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233066.929814:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233066/real 1761233066] req@0000000053e52331 x1846780895109632/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233091 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233066.929822:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000053e52331 x1846780895109632/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233091 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233066.929827:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233066.929829:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233066.929831:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895109632:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233066.929835:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895109696:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233066.929847:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233066.929852:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233066/real 1761233066] req@00000000880f16b4 x1846780895109696/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233091 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233066.929858:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000880f16b4 x1846780895109696/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233091 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233066.929864:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233066.929865:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233066.929867:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895109696:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233066.929870:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895109760:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233066.929881:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233066.929885:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233066/real 1761233066] req@00000000cb5be845 x1846780895109760/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233091 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233066.929891:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cb5be845 x1846780895109760/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233091 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233066.929896:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233066.929897:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233066.929899:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895109760:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233066.929902:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895109824:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233066.929912:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233066.929915:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233066/real 1761233066] req@00000000a95774fb x1846780895109824/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233091 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233066.929922:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a95774fb x1846780895109824/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233091 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233066.929927:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233066.929928:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233066.929929:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895109824:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233066.929933:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895109888:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233066.929943:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233066.929947:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233066/real 1761233066] req@000000000c736bb2 x1846780895109888/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233091 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233066.929953:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000000c736bb2 x1846780895109888/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233091 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233066.929957:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233066.929958:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233066.929960:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895109888:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233066.929963:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895109952:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233066.929973:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233066.929977:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233066/real 1761233066] req@00000000038209f5 x1846780895109952/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233091 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233066.929982:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000038209f5 x1846780895109952/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233091 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233066.929987:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233066.929988:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233066.929990:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895109952:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233069.745382:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233070.769495:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233072.049335:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233072.049344:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233072.049347:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233072.049356:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5632 00000100:00080000:0.0:1761233072.049359:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233072.049365:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233072.049396:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895110016/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233072.049417:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233072.049420:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233072.049421:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233072.049423:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5632 00000100:00080000:0.0:1761233072.049425:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233072.049427:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233072.049436:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233072.049444:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895110080/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761233072.049450:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895110016:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233072.049454:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233072.049457:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233072.049458:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233072.049460:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5632 00000100:00080000:0.0:1761233072.049461:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233072.049463:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233072.049466:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233072.049475:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895110144/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233072.049480:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233072.049482:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233072.049483:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233072.049486:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5632 00000100:00080000:0.0:1761233072.049487:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233072.049489:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233072.049491:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233072.049497:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895110208/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233072.049501:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233072.049504:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233072.049504:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233072.049507:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5632 00000100:00080000:0.0:1761233072.049508:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233072.049511:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233072.049514:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1761233072.049516:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233072.049523:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895110272/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233072.049534:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233072.049536:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233072.049537:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233072.049540:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5632 00000100:00080000:0.0:1761233072.049542:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233072.049545:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233072.049547:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233072.049549:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233072/real 1761233072] req@000000001f9ec359 x1846780895110016/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233097 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233072.049555:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895110336/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233072.049568:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233072.049570:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233072.049571:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1761233072.049573:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895110016/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233097 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233072.049575:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5632 00000100:00080000:0.0:1761233072.049579:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233072.049584:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233072.049589:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1761233072.049593:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233072.049597:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233072.049599:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895110400/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761233072.049603:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895110016:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761233072.049631:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895110080:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233072.049647:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233072.049652:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233072/real 1761233072] req@000000004130b6bd x1846780895110080/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233097 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233072.049659:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895110080/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233097 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233072.049663:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233072.049665:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233072.049666:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895110080:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233072.049669:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895110144:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233072.049707:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233072.049712:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233072/real 1761233072] req@00000000ef655201 x1846780895110144/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233097 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233072.049719:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895110144/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233097 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233072.049734:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233072.049735:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233072.049737:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895110144:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233072.049740:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895110208:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233072.049751:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233072.049755:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233072/real 1761233072] req@000000006d61dd06 x1846780895110208/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233097 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233072.049760:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895110208/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233097 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233072.049765:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233072.049766:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233072.049767:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895110208:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233072.049770:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895110272:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233072.049779:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233072.049783:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233072/real 1761233072] req@00000000efa29eb9 x1846780895110272/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233097 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233072.049789:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895110272/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233097 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233072.049793:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233072.049794:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233072.049796:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895110272:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233072.049799:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895110336:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233072.049810:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233072.049813:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233072/real 1761233072] req@000000001db96463 x1846780895110336/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233097 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233072.049820:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895110336/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233097 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233072.049824:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233072.049825:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233072.049827:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895110336:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233072.049830:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895110400:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233072.049838:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233072.049842:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233072/real 1761233072] req@000000007bd1ec64 x1846780895110400/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233097 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233072.049847:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895110400/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233097 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233072.049851:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233072.049852:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233072.049854:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895110400:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233073.841368:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233074.865491:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1761233077.169575:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895110464:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761233077.169722:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895110528:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233077.169767:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895110592:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233077.169808:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895110656:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233077.169852:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895110720:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233077.169908:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895110784:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233077.169950:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895110848:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233077.937463:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233078.961350:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1761233080.241956:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233077/real 1761233080] req@0000000089b8c4f1 x1846780895110784/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233102 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233080.241992:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895110784/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233102 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233080.242003:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233080.242007:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233080.242011:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895110784:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233080.242018:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233077/real 1761233080] req@00000000829a5595 x1846780895110848/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233102 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233080.242027:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895110848/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233102 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233080.242033:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233080.242035:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233080.242038:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895110848:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233080.242043:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233077/real 1761233080] req@000000003b40830d x1846780895110592/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233102 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233080.242051:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895110592/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233102 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233080.242059:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233080.242061:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233080.242063:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895110592:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233080.242069:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233077/real 1761233080] req@000000002a0ba65f x1846780895110656/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233102 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233080.242082:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895110656/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233102 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233080.242092:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233080.242095:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233080.242099:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895110656:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233080.242114:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233077/real 1761233080] req@0000000057e55ffa x1846780895110720/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233102 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233080.242131:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895110720/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233102 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233080.242142:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233080.242146:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233080.242150:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895110720:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233080.242157:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233077/real 1761233080] req@000000007aee79ba x1846780895110464/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233102 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233080.242173:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895110464/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233102 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233080.242192:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233080.242196:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233080.242237:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895110464:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761233080.242248:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233077/real 1761233080] req@000000009706a782 x1846780895110528/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233102 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233080.242265:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895110528/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233102 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233080.242275:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233080.242279:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233080.242284:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895110528:192.168.201.157@tcp:38:kworker/u8:3.0 00010000:00010000:1.0:1761233082.033457:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1761233082.289435:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895110912:192.168.201.157@tcp:250:kworker/u8:3.0 00000800:00000100:1.0:1761233082.289503:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233082.289549:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233082/real 1761233082] req@00000000c9993cd4 x1846780895110912/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233107 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233082.289569:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895110912/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233107 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233082.289582:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233082.289586:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233082.289592:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895110912:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233082.289612:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895110976:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233082.289638:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233082.289652:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233082/real 1761233082] req@000000002625ec0b x1846780895110976/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233107 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233082.289664:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895110976/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233107 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233082.289671:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233082.289673:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233082.289676:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895110976:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233082.289703:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895111040:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233082.289723:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233082.289730:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233082/real 1761233082] req@00000000bbf6af7b x1846780895111040/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233107 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233082.289741:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895111040/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233107 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233082.289748:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233082.289750:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233082.289753:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895111040:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233082.289758:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895111104:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233082.289777:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233082.289785:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233082/real 1761233082] req@000000009706a782 x1846780895111104/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233107 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233082.289796:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895111104/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233107 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233082.289803:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233082.289806:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233082.289808:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895111104:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233082.289813:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895111168:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233082.289829:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233082.289836:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233082/real 1761233082] req@000000007aee79ba x1846780895111168/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233107 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233082.289846:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895111168/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233107 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233082.289853:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233082.289855:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233082.289858:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895111168:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233082.289863:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895111232:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233082.289880:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233082.289887:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233082/real 1761233082] req@0000000057e55ffa x1846780895111232/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233107 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233082.289897:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895111232/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233107 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233082.289904:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233082.289906:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233082.289909:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895111232:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233082.289914:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895111296:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233082.289932:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233082.289939:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233082/real 1761233082] req@000000002a0ba65f x1846780895111296/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233107 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233082.289950:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895111296/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233107 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233082.289957:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233082.289962:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233082.289964:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895111296:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233083.057459:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233086.129545:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233087.153484:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233087.409412:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233087.409429:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233087.409453:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233087.409470:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5648 00000100:00080000:2.0:1761233087.409477:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233087.409499:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233087.409557:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895111360/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233087.409608:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233087.409615:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233087.409618:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233087.409624:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5648 00000100:00080000:2.0:1761233087.409628:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233087.409639:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233087.409654:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233087.409663:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895111360:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761233087.409680:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895111424/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233087.409696:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233087.409700:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233087.409702:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233087.409707:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5648 00000100:00080000:2.0:1761233087.409710:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233087.409714:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233087.409721:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233087.409733:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895111488/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233087.409743:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233087.409747:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233087.409750:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1761233087.409752:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233087.409758:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5648 00000100:00080000:2.0:1761233087.409761:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233087.409765:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233087.409772:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233087.409785:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895111552/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233087.409800:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233087/real 1761233087] req@000000004f29ca26 x1846780895111360/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233112 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233087.409807:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233087.409819:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233087.409822:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233087.409833:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5648 00000100:00080000:2.0:1761233087.409836:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761233087.409843:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895111360/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233112 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233087.409844:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233087.409854:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1761233087.409869:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1761233087.409870:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895111616/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233087.409877:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233087.409888:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895111360:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761233087.409901:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233087.409911:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233087.409914:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233087.409920:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5648 00000100:00080000:2.0:1761233087.409922:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233087.409926:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233087.409930:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233087.409943:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895111424:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233087.409945:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895111680/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233087.409964:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233087.409968:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233087.409971:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233087.409975:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5648 00000100:00080000:2.0:1761233087.409978:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1761233087.409985:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233087.409987:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233087.409995:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233087.410004:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233087/real 1761233087] req@0000000033765045 x1846780895111424/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233112 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233087.410012:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895111744/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233087.410037:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895111424/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233112 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233087.410049:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233087.410053:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233087.410060:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895111424:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233087.410068:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895111488:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233087.410090:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233087.410097:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233087/real 1761233087] req@000000009d315fa5 x1846780895111488/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233112 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233087.410109:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895111488/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233112 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233087.410117:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233087.410119:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233087.410122:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895111488:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233087.410128:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895111552:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233087.410166:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233087.410180:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233087/real 1761233087] req@0000000040d5f361 x1846780895111552/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233112 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233087.410199:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895111552/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233112 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233087.410248:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233087.410250:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233087.410255:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895111552:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233087.410262:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895111616:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233087.410290:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233087.410299:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233087/real 1761233087] req@000000007dcf84ae x1846780895111616/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233112 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233087.410312:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895111616/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233112 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233087.410320:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233087.410322:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233087.410352:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895111616:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233087.410380:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895111680:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233087.410410:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233087.410418:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233087/real 1761233087] req@000000007091f547 x1846780895111680/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233112 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233087.410430:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895111680/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233112 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233087.410475:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233087.410478:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233087.410482:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895111680:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233087.410489:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895111744:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233087.410511:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233087.410520:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233087/real 1761233087] req@0000000064a2cea0 x1846780895111744/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233112 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233087.410532:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895111744/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233112 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233087.410540:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233087.410544:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233087.410547:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895111744:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233090.225482:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233091.249444:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233092.529323:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233092.529335:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233092.529339:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233092.529350:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5653 00000100:00080000:0.0:1761233092.529385:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233092.529393:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233092.529428:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895111808/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233092.529457:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233092.529461:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233092.529463:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233092.529467:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5653 00000100:00080000:0.0:1761233092.529470:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233092.529475:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233092.529484:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233092.529498:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895111872/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233092.529509:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233092.529512:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895111808:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233092.529515:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233092.529518:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233092.529522:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5653 00000100:00080000:0.0:1761233092.529524:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233092.529528:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233092.529535:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233092.529546:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895111936/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233092.529555:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233092.529558:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233092.529560:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233092.529563:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5653 00000100:00080000:0.0:1761233092.529566:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233092.529569:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233092.529573:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761233092.529576:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233092.529587:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895112000/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233092.529597:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233092/real 1761233092] req@000000005593c994 x1846780895111808/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233117 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233092.529598:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233092.529606:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233092.529608:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233092.529615:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5653 00000100:00080000:0.0:1761233092.529617:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761233092.529620:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895111808/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233117 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233092.529624:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233092.529633:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1761233092.529633:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233092.529637:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233092.529641:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895111808:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233092.529648:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895112064/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233092.529658:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233092.529662:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1761233092.529664:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895111872:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:0.0:1761233092.529665:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233092.529673:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5653 00000100:00080000:0.0:1761233092.529676:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233092.529680:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:1.0:1761233092.529684:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:0.0:1761233092.529689:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233092.529693:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233092/real 1761233092] req@00000000c6e06344 x1846780895111872/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233117 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233092.529706:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895112128/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233092.529706:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895111872/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233117 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233092.529733:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233092.529737:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233092.529738:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233092.529741:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895111872:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:0.0:1761233092.529747:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233092.529750:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761233092.529751:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895111936:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00080000:0.0:1761233092.529758:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5653 00000100:00080000:0.0:1761233092.529762:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233092.529766:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:1.0:1761233092.529770:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:0.0:1761233092.529773:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233092.529777:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233092/real 1761233092] req@000000004862b808 x1846780895111936/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233117 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233092.529786:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895111936/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233117 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233092.529790:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895112192/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233092.529796:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233092.529800:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233092.529806:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895111936:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233092.529813:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895112000:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233092.529826:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233092.529830:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233092/real 1761233092] req@000000006f2efc8b x1846780895112000/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233117 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233092.529837:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895112000/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233117 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233092.529841:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233092.529843:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233092.529844:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895112000:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233092.529847:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895112064:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233092.529856:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233092.529860:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233092/real 1761233092] req@00000000eca9ddc2 x1846780895112064/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233117 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233092.529867:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895112064/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233117 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233092.529871:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233092.529872:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233092.529874:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895112064:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233092.529885:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895112128:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233092.529894:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233092.529898:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233092/real 1761233092] req@0000000034d49f59 x1846780895112128/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233117 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233092.529904:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895112128/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233117 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233092.529908:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233092.529909:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233092.529911:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895112128:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233092.529913:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895112192:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233092.529922:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233092.529926:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233092/real 1761233092] req@000000005beaae31 x1846780895112192/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233117 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233092.529931:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895112192/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233117 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233092.529935:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233092.529937:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233092.529939:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895112192:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233094.321447:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233095.345386:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233097.649328:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233097.649341:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233097.649346:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233097.649358:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5658 00000100:00080000:2.0:1761233097.649362:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233097.649371:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233097.649410:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895112256/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233097.649439:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233097.649443:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233097.649445:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233097.649448:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5658 00000100:00080000:2.0:1761233097.649451:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233097.649455:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233097.649465:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233097.649484:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895112256:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761233097.649485:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895112320/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233097.649499:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233097.649503:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233097.649504:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233097.649508:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5658 00000100:00080000:2.0:1761233097.649511:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233097.649516:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233097.649524:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233097.649536:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895112384/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761233097.649546:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233097.649547:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233097.649552:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233097.649554:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233097.649559:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5658 00000100:00080000:2.0:1761233097.649561:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761233097.649567:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233097/real 1761233097] req@0000000064a2cea0 x1846780895112256/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233122 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233097.649567:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233097.649574:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233097.649584:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895112256/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233122 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233097.649590:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895112448/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233097.649595:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233097.649599:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233097.649605:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895112256:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761233097.649606:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233097.649612:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233097.649614:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233097.649619:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5658 00000100:00080000:2.0:1761233097.649621:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233097.649626:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761233097.649629:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895112320:192.168.201.157@tcp:38:kworker/u8:3.0 00000008:00100000:2.0:1761233097.649630:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233097.649647:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895112512/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761233097.649653:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233097.649661:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233097/real 1761233097] req@000000007091f547 x1846780895112320/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233122 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233097.649662:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233097.649672:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233097.649675:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761233097.649677:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895112320/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233122 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233097.649682:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5658 00000100:00080000:1.0:1761233097.649687:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233097.649688:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233097.649692:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761233097.649695:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761233097.649698:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895112320:192.168.201.157@tcp:38:kworker/u8:3.0 00000008:00100000:2.0:1761233097.649704:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233097.649707:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895112384:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233097.649720:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895112576/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761233097.649726:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233097.649733:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233097/real 1761233097] req@000000007dcf84ae x1846780895112384/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233122 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233097.649734:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233097.649743:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233097.649747:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761233097.649750:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895112384/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233122 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233097.649756:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5658 00000100:00080000:1.0:1761233097.649758:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233097.649761:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233097.649762:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233097.649767:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895112384:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761233097.649770:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761233097.649775:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895112448:192.168.201.157@tcp:8:kworker/u8:3.0 00000008:00100000:2.0:1761233097.649777:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761233097.649789:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233097.649795:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895112640/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233097.649797:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233097/real 1761233097] req@0000000040d5f361 x1846780895112448/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233122 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233097.649823:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895112448/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233122 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233097.649829:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233097.649830:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233097.649832:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895112448:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233097.649860:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895112512:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233097.649872:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233097.649876:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233097/real 1761233097] req@000000009d315fa5 x1846780895112512/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233122 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233097.649883:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895112512/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233122 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233097.649887:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233097.649889:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233097.649891:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895112512:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233097.649893:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895112576:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233097.649902:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233097.649905:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233097/real 1761233097] req@0000000033765045 x1846780895112576/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233122 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233097.649911:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895112576/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233122 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233097.649915:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233097.649917:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233097.649918:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895112576:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233097.649921:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895112640:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233097.649930:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233097.649933:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233097/real 1761233097] req@000000004f29ca26 x1846780895112640/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233122 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233097.649938:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895112640/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233122 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233097.649943:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233097.649944:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233097.649946:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895112640:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233098.417476:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233099.441457:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233102.513414:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1761233102.769596:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895112704:192.168.201.157@tcp:250:kworker/u8:3.0 00000800:00000100:1.0:1761233102.769693:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233102.769719:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233102/real 1761233102] req@000000003b40830d x1846780895112704/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233127 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233102.769735:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895112704/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233127 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233102.769743:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233102.769746:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233102.769750:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895112704:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233102.769754:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895112768:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233102.769769:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233102.769773:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233102/real 1761233102] req@00000000829a5595 x1846780895112768/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233127 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233102.769780:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895112768/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233127 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233102.769785:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233102.769786:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233102.769788:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895112768:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233102.769792:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895112832:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233102.769804:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233102.769808:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233102/real 1761233102] req@0000000089b8c4f1 x1846780895112832/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233127 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233102.769814:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895112832/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233127 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233102.769819:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233102.769820:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233102.769822:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895112832:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233102.769845:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895112896:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233102.769855:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233102.769859:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233102/real 1761233102] req@0000000070fa3bd6 x1846780895112896/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233127 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233102.769865:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895112896/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233127 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233102.769870:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233102.769871:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233102.769873:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895112896:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233102.769876:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895112960:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233102.769885:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233102.769889:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233102/real 1761233102] req@0000000090702bc4 x1846780895112960/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233127 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233102.769896:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895112960/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233127 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233102.769901:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233102.769902:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233102.769903:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895112960:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233102.769906:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895113024:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233102.769925:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233102.769929:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233102/real 1761233102] req@00000000b6fdd7f9 x1846780895113024/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233127 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233102.769934:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895113024/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233127 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233102.769938:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233102.769939:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233102.769942:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895113024:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233102.769945:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895113088:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233102.769955:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233102.769958:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233102/real 1761233102] req@000000008e58c4b8 x1846780895113088/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233127 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233102.769964:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895113088/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233127 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233102.769969:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233102.769971:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233102.769973:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895113088:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233103.537419:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233106.609482:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233107.633517:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233107.889333:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233107.889342:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233107.889345:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233107.889355:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5668 00000100:00080000:0.0:1761233107.889358:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233107.889364:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233107.889397:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000078e7052d x1846780895113152/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233107.889416:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233107.889419:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233107.889420:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233107.889423:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5668 00000100:00080000:0.0:1761233107.889424:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233107.889426:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233107.889433:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233107.889441:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000addc3db8 x1846780895113216/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233107.889449:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233107.889449:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895113152:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233107.889455:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233107.889458:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233107.889461:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5668 00000100:00080000:0.0:1761233107.889462:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233107.889464:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233107.889467:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233107.889476:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895113280/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233107.889481:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233107.889483:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233107.889484:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233107.889486:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5668 00000100:00080000:0.0:1761233107.889487:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233107.889489:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233107.889492:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233107.889498:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895113344/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233107.889503:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233107.889505:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233107.889506:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233107.889508:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5668 00000100:00080000:0.0:1761233107.889509:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233107.889512:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233107.889515:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761233107.889516:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233107.889524:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895113408/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233107.889529:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233107.889532:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233107.889533:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761233107.889536:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233107/real 1761233107] req@0000000078e7052d x1846780895113152/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233132 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233107.889537:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5668 00000100:00080000:0.0:1761233107.889541:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233107.889546:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233107.889551:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233107.889560:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895113472/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233107.889562:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000078e7052d x1846780895113152/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233132 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233107.889578:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233107.889581:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233107.889582:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233107.889583:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233107.889587:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233107.889590:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5668 00000100:00100000:1.0:1761233107.889593:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895113152:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233107.889594:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233107.889600:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233107.889603:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233107.889609:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895113536/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233107.889621:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895113216:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233107.889639:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233107.889648:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233107/real 1761233107] req@00000000addc3db8 x1846780895113216/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233132 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233107.889655:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000addc3db8 x1846780895113216/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233132 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233107.889660:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233107.889662:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233107.889664:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895113216:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233107.889668:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895113280:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233107.889678:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233107.889682:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233107/real 1761233107] req@000000005beaae31 x1846780895113280/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233132 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233107.889688:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895113280/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233132 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233107.889693:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233107.889694:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233107.889695:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895113280:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233107.889698:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895113344:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233107.889707:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233107.889711:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233107/real 1761233107] req@0000000034d49f59 x1846780895113344/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233132 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233107.889716:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895113344/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233132 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233107.889720:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233107.889721:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233107.889723:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895113344:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233107.889726:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895113408:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233107.889735:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233107.889738:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233107/real 1761233107] req@00000000eca9ddc2 x1846780895113408/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233132 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233107.889745:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895113408/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233132 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233107.889750:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233107.889751:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233107.889752:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895113408:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233107.889755:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895113472:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233107.889764:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233107.889768:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233107/real 1761233107] req@000000006f2efc8b x1846780895113472/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233132 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233107.889773:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895113472/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233132 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233107.889778:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233107.889779:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233107.889780:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895113472:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233107.889783:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895113536:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233107.889792:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233107.889795:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233107/real 1761233107] req@000000004862b808 x1846780895113536/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233132 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233107.889801:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895113536/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233132 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233107.889805:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233107.889807:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233107.889808:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895113536:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233110.705437:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233111.729493:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233113.009390:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233113.009406:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233113.009413:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233113.009429:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5673 00000100:00080000:1.0:1761233113.009434:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233113.009444:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233113.009503:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a611ee34 x1846780895113600/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233113.009544:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233113.009548:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233113.009549:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233113.009553:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5673 00000100:00080000:1.0:1761233113.009555:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233113.009559:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233113.009571:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233113.009587:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000328368c8 x1846780895113664/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233113.009597:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233113.009602:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233113.009603:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233113.009607:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5673 00000100:00080000:1.0:1761233113.009611:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233113.009614:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233113.009619:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233113.009629:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895113600:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761233113.009631:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c652d0ad x1846780895113728/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233113.009662:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233113.009664:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233113.009666:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233113.009669:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5673 00000100:00080000:1.0:1761233113.009671:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233113.009674:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233113.009677:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233113.009692:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ba3340f x1846780895113792/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233113.009699:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233113.009703:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233113.009704:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233113.009708:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5673 00000100:00080000:1.0:1761233113.009710:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233113.009715:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233113.009720:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233113.009729:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d74b2e3c x1846780895113856/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233113.009737:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233113.009742:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233113.009743:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233113.009747:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5673 00000100:00080000:1.0:1761233113.009750:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233113.009753:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233113.009756:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1761233113.009766:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233113.009768:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001aedf9b1 x1846780895113920/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233113.009778:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233113.009781:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233113.009783:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233113.009790:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5673 00000100:00080000:1.0:1761233113.009793:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233113.009798:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233113.009803:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233113.009823:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000568f2a96 x1846780895113984/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761233113.009823:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233113/real 1761233113] req@00000000a611ee34 x1846780895113600/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233138 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233113.009908:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a611ee34 x1846780895113600/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233138 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233113.009929:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233113.009934:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233113.009941:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895113600:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761233113.009957:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895113664:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233113.010000:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233113.010015:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233113/real 1761233113] req@00000000328368c8 x1846780895113664/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233138 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233113.010065:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000328368c8 x1846780895113664/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233138 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233113.010080:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233113.010085:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233113.010090:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895113664:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233113.010136:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895113728:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233113.010179:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233113.010195:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233113/real 1761233113] req@00000000c652d0ad x1846780895113728/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233138 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233113.010258:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c652d0ad x1846780895113728/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233138 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233113.010272:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233113.010276:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233113.010282:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895113728:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233113.010293:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895113792:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233113.010331:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233113.010345:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233113/real 1761233113] req@000000007ba3340f x1846780895113792/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233138 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233113.010365:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007ba3340f x1846780895113792/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233138 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233113.010378:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233113.010382:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233113.010388:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895113792:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233113.010400:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895113856:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233113.010433:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233113.010446:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233113/real 1761233113] req@00000000d74b2e3c x1846780895113856/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233138 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233113.010464:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d74b2e3c x1846780895113856/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233138 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233113.010477:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233113.010481:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233113.010487:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895113856:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233113.010497:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895113920:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233113.010532:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233113.010546:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233113/real 1761233113] req@000000001aedf9b1 x1846780895113920/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233138 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233113.010564:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001aedf9b1 x1846780895113920/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233138 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233113.010577:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233113.010581:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233113.010587:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895113920:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233113.010597:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895113984:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233113.010632:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233113.010647:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233113/real 1761233113] req@00000000568f2a96 x1846780895113984/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233138 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233113.010665:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000568f2a96 x1846780895113984/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233138 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233113.010676:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233113.010680:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233113.010685:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895113984:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233114.801349:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233115.825390:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233118.129429:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233118.129454:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233118.129459:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233118.129476:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5679 00000100:00080000:1.0:1761233118.129482:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233118.129494:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233118.129543:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090800dde x1846780895114048/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233118.129574:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233118.129578:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233118.129580:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233118.129584:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5679 00000100:00080000:1.0:1761233118.129586:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233118.129591:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233118.129603:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233118.129625:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e467e107 x1846780895114112/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761233118.129632:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090800dde pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895114048:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:1.0:1761233118.129642:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233118.129647:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233118.129649:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233118.129654:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5679 00000100:00080000:1.0:1761233118.129656:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233118.129659:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233118.129664:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233118.129679:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cbad49de x1846780895114176/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233118.129688:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233118.129691:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233118.129692:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233118.129696:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5679 00000100:00080000:1.0:1761233118.129699:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233118.129702:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233118.129708:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233118.129719:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000568f2a96 x1846780895114240/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:2.0:1761233118.129728:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:1.0:1761233118.129737:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233118.129742:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233118.129744:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233118.129750:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5679 00000100:00080000:1.0:1761233118.129753:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233118.129758:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233118.129763:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233118.129764:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233118/real 1761233118] req@0000000090800dde x1846780895114048/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233143 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233118.129778:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001aedf9b1 x1846780895114304/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233118.129789:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1761233118.129790:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090800dde x1846780895114048/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233143 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233118.129798:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233118.129802:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233118.129812:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5679 00000100:00080000:2.0:1761233118.129814:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233118.129816:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233118.129820:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:1.0:1761233118.129825:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:2.0:1761233118.129830:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090800dde pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895114048:192.168.201.157@tcp:250:kworker/u8:2.0 00000008:00100000:1.0:1761233118.129834:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233118.129850:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d74b2e3c x1846780895114368/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233118.129858:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233118.129862:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233118.129865:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1761233118.129866:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e467e107 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895114112:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:1.0:1761233118.129877:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5679 00000100:00080000:1.0:1761233118.129883:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233118.129887:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233118.129892:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233118.129904:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ba3340f x1846780895114432/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:2.0:1761233118.129911:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233118.129929:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233118/real 1761233118] req@00000000e467e107 x1846780895114112/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233143 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233118.129945:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e467e107 x1846780895114112/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233143 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233118.129992:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233118.129995:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233118.130016:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e467e107 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895114112:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233118.130023:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbad49de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895114176:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233118.130046:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233118.130054:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233118/real 1761233118] req@00000000cbad49de x1846780895114176/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233143 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233118.130064:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cbad49de x1846780895114176/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233143 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233118.130071:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233118.130073:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233118.130075:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbad49de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895114176:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233118.130080:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895114240:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233118.130094:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233118.130100:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233118/real 1761233118] req@00000000568f2a96 x1846780895114240/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233143 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233118.130109:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000568f2a96 x1846780895114240/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233143 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233118.130117:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233118.130119:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233118.130121:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895114240:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233118.130125:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895114304:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233118.130140:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233118.130146:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233118/real 1761233118] req@000000001aedf9b1 x1846780895114304/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233143 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233118.130162:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001aedf9b1 x1846780895114304/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233143 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233118.130168:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233118.130170:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233118.130173:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895114304:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233118.130177:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895114368:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233118.130191:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233118.130197:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233118/real 1761233118] req@00000000d74b2e3c x1846780895114368/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233143 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233118.130207:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d74b2e3c x1846780895114368/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233143 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233118.130213:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233118.130252:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233118.130255:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895114368:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233118.130279:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895114432:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233118.130300:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233118.130307:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233118/real 1761233118] req@000000007ba3340f x1846780895114432/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233143 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233118.130318:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007ba3340f x1846780895114432/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233143 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233118.130325:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233118.130327:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233118.130329:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895114432:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233118.897478:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233119.921468:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233122.993421:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1761233123.249462:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895114496:192.168.201.157@tcp:250:kworker/u8:3.0 00000800:00000100:2.0:1761233123.249527:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233123.249547:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233123/real 1761233123] req@00000000e99e5a12 x1846780895114496/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233148 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233123.249562:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895114496/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233148 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233123.249573:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233123.249575:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233123.249579:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895114496:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761233123.249602:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895114560:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761233123.249622:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233123.249629:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233123/real 1761233123] req@000000007adf99ab x1846780895114560/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233148 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233123.249646:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895114560/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233148 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233123.249654:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233123.249656:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233123.249659:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895114560:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233123.249665:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895114624:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233123.249682:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233123.249687:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233123/real 1761233123] req@0000000080131ea7 x1846780895114624/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233148 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233123.249693:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895114624/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233148 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233123.249698:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233123.249699:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233123.249701:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895114624:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233123.249704:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895114688:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233123.249713:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233123.249717:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233123/real 1761233123] req@000000008e58c4b8 x1846780895114688/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233148 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233123.249722:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895114688/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233148 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233123.249727:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233123.249729:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233123.249730:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895114688:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233123.249733:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895114752:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761233123.249743:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233123.249746:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233123/real 1761233123] req@00000000b6fdd7f9 x1846780895114752/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233148 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233123.249752:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895114752/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233148 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233123.249756:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233123.249757:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233123.249759:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895114752:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233123.249762:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895114816:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233123.249770:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233123.249774:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233123/real 1761233123] req@0000000090702bc4 x1846780895114816/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233148 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233123.249779:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895114816/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233148 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233123.249784:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233123.249785:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233123.249787:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895114816:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233123.249801:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895114880:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233123.249813:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233123.249817:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233123/real 1761233123] req@0000000070fa3bd6 x1846780895114880/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233148 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233123.249823:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895114880/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233148 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233123.249827:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233123.249828:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233123.249830:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895114880:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233124.017412:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233127.089353:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233128.113478:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1761233128.369574:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895114944:192.168.201.157@tcp:250:kworker/u8:3.0 00000800:00000100:2.0:1761233128.369687:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233128.369728:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233128/real 1761233128] req@0000000089b8c4f1 x1846780895114944/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233153 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233128.369761:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895114944/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233153 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233128.369828:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233128.369834:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233128.369846:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895114944:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761233128.369867:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895115008:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761233128.369909:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233128.369924:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233128/real 1761233128] req@00000000829a5595 x1846780895115008/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233153 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233128.369940:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895115008/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233153 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233128.369950:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233128.369953:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233128.369957:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895115008:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233128.369990:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895115072:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233128.370012:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233128.370020:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233128/real 1761233128] req@000000003b40830d x1846780895115072/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233153 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233128.370032:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895115072/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233153 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233128.370040:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233128.370042:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233128.370046:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895115072:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233128.370052:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895115136:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233128.370070:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233128.370077:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233128/real 1761233128] req@000000002a0ba65f x1846780895115136/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233153 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233128.370089:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895115136/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233153 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233128.370097:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233128.370099:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233128.370102:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895115136:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233128.370109:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895115200:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761233128.370128:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233128.370135:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233128/real 1761233128] req@0000000057e55ffa x1846780895115200/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233153 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233128.370147:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895115200/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233153 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233128.370156:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233128.370159:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233128.370163:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895115200:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233128.370173:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895115264:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233128.370261:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233128.370279:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233128/real 1761233128] req@000000007aee79ba x1846780895115264/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233153 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233128.370301:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895115264/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233153 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233128.370317:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233128.370321:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233128.370327:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895115264:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233128.370336:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895115328:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233128.370365:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233128.370376:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233128/real 1761233128] req@000000009706a782 x1846780895115328/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233153 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233128.370389:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895115328/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233153 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233128.370398:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233128.370401:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233128.370404:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895115328:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233131.185452:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233132.209446:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1761233133.489494:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895115392:192.168.201.157@tcp:250:kworker/u8:3.0 00000800:00000100:2.0:1761233133.489559:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233133.489581:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233133/real 1761233133] req@00000000bbf6af7b x1846780895115392/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233158 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233133.489597:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895115392/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233158 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233133.489606:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233133.489611:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233133.489617:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895115392:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761233133.489640:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895115456:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761233133.489662:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233133.489670:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233133/real 1761233133] req@000000002625ec0b x1846780895115456/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233158 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233133.489702:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895115456/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233158 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233133.489727:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233133.489729:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233133.489733:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895115456:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233133.489741:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895115520:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233133.489759:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233133.489766:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233133/real 1761233133] req@00000000c9993cd4 x1846780895115520/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233158 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233133.489774:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895115520/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233158 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233133.489782:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233133.489787:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233133.489794:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895115520:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233133.489803:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895115584:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233133.489822:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233133.489829:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233133/real 1761233133] req@000000009706a782 x1846780895115584/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233158 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233133.489845:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895115584/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233158 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233133.489852:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233133.489854:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233133.489860:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895115584:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233133.489881:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895115648:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761233133.489905:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233133.489912:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233133/real 1761233133] req@000000007aee79ba x1846780895115648/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233158 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233133.489923:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895115648/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233158 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233133.489931:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233133.489933:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233133.489936:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895115648:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233133.489942:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895115712:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233133.489964:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233133.489972:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233133/real 1761233133] req@0000000057e55ffa x1846780895115712/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233158 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233133.489981:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895115712/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233158 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233133.489986:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233133.489988:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233133.489989:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895115712:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233133.489993:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895115776:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233133.490006:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233133.490010:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233133/real 1761233133] req@000000002a0ba65f x1846780895115776/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233158 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233133.490017:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895115776/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233158 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233133.490022:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233133.490023:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233133.490025:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895115776:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233135.281415:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233136.305466:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233138.609364:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233138.609379:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233138.609385:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233138.609399:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5699 00000100:00080000:2.0:1761233138.609405:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233138.609414:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233138.609458:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895115840/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233138.609494:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233138.609499:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233138.609501:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233138.609506:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5699 00000100:00080000:2.0:1761233138.609509:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233138.609514:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233138.609527:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233138.609548:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895115840:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761233138.609548:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895115904/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233138.609570:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233138.609574:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233138.609577:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233138.609584:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5699 00000100:00080000:2.0:1761233138.609587:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233138.609592:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233138.609600:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233138.609616:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895115968/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233138.609628:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233138.609654:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233138.609659:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1761233138.609660:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233138.609670:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5699 00000100:00080000:2.0:1761233138.609674:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233138.609680:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233138.609688:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233138.609695:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233138/real 1761233138] req@00000000fd3935a6 x1846780895115840/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233163 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233138.609708:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895116032/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233138.609725:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233138.609728:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895115840/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233163 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233138.609734:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233138.609738:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233138.609750:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5699 00000100:00080000:1.0:1761233138.609753:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233138.609755:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233138.609762:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761233138.609765:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761233138.609773:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895115840:192.168.201.157@tcp:250:kworker/u8:3.0 00000008:00100000:2.0:1761233138.609774:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233138.609797:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895116096/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233138.609814:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895115904:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:2.0:1761233138.609838:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233138.609845:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233138.609848:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233138.609856:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5699 00000800:00000100:1.0:1761233138.609862:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233138.609865:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233138.609876:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761233138.609881:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233138/real 1761233138] req@00000000135e9a6e x1846780895115904/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233163 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:2.0:1761233138.609888:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233138.609904:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895115904/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233163 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233138.609918:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895116160/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233138.609920:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233138.609926:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233138.609936:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895115904:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:2.0:1761233138.609952:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233138.609955:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895115968:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761233138.609965:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233138.609969:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233138.609978:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5699 00000100:00080000:2.0:1761233138.609982:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233138.609989:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233138.609997:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761233138.609998:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233138.610014:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233138/real 1761233138] req@000000006ff1fdbc x1846780895115968/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233163 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233138.610025:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895116224/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233138.610037:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895115968/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233163 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233138.610052:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233138.610056:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233138.610061:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895115968:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233138.610073:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895116032:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233138.610102:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233138.610115:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233138/real 1761233138] req@0000000088939481 x1846780895116032/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233163 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233138.610130:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895116032/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233163 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233138.610141:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233138.610144:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233138.610148:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895116032:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233138.610176:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895116096:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233138.610236:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233138.610250:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233138/real 1761233138] req@000000006d8c3eb0 x1846780895116096/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233163 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233138.610264:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895116096/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233163 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233138.610275:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233138.610278:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233138.610282:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895116096:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233138.610290:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895116160:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233138.610339:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233138.610351:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233138/real 1761233138] req@000000006c1041ba x1846780895116160/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233163 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233138.610373:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895116160/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233163 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233138.610393:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233138.610396:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233138.610401:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895116160:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233138.610409:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895116224:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233138.610438:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233138.610449:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233138/real 1761233138] req@00000000fd4aa3a2 x1846780895116224/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233163 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233138.610464:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895116224/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233163 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233138.610473:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233138.610477:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233138.610481:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895116224:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233139.377456:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233140.401533:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233143.473435:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233143.729349:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233143.729360:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233143.729365:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233143.729378:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5704 00000100:00080000:2.0:1761233143.729382:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233143.729391:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233143.729430:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895116288/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233143.729460:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233143.729464:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233143.729465:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233143.729469:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5704 00000100:00080000:2.0:1761233143.729471:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233143.729474:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233143.729483:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233143.729496:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895116352/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233143.729524:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233143.729528:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233143.729530:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233143.729535:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5704 00000100:00080000:2.0:1761233143.729537:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233143.729541:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233143.729587:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233143.729604:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895116416/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233143.729614:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233143.729618:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233143.729620:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233143.729624:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5704 00000100:00080000:2.0:1761233143.729627:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233143.729630:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233143.729652:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233143.729668:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895116480/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233143.729678:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233143.729683:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233143.729685:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233143.729689:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5704 00000100:00080000:2.0:1761233143.729692:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233143.729696:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233143.729703:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233143.729715:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895116544/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233143.729724:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233143.729728:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233143.729730:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233143.729734:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5704 00000100:00080000:2.0:1761233143.729737:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233143.729744:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233143.729753:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233143.729768:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895116608/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233143.729785:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233143.729789:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233143.729791:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233143.729796:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5704 00000100:00080000:2.0:1761233143.729799:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233143.729804:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233143.729810:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233143.729822:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895116672/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:1.0:1761233144.497504:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233145.521470:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1761233146.801729:0:2209:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:0.0:1761233146.801794:0:2209:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1761233146.801816:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233146.801886:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233146.801898:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233146.801910:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233146.801921:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233146.801932:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233146.801941:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00010000:00010000:1.0:1761233147.569422:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233148.593397:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233148.849426:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233148.849437:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233148.849441:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233148.849452:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5709 00000100:00080000:1.0:1761233148.849456:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233148.849464:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233148.849519:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c652d0ad x1846780895116736/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233148.849546:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233148.849549:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233148.849551:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233148.849555:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5709 00000100:00080000:1.0:1761233148.849557:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233148.849560:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233148.849569:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233148.849580:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000328368c8 x1846780895116800/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233148.849590:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233148.849593:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233148.849595:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233148.849598:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5709 00000100:00080000:1.0:1761233148.849600:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233148.849603:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233148.849607:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233148.849616:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a611ee34 x1846780895116864/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233148.849625:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233148.849629:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233148.849632:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233148.849643:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5709 00000100:00080000:1.0:1761233148.849646:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233148.849648:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233148.849652:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233148.849661:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000038209f5 x1846780895116928/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233148.849669:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233148.849672:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233148.849673:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233148.849676:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5709 00000100:00080000:1.0:1761233148.849678:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233148.849681:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233148.849687:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233148.849702:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000c736bb2 x1846780895116992/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233148.849713:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233148.849718:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233148.849720:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233148.849725:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5709 00000100:00080000:1.0:1761233148.849728:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233148.849738:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233148.849746:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233148.849766:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a95774fb x1846780895117056/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233148.849810:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233148.849817:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233148.849821:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233148.849830:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5709 00000100:00080000:1.0:1761233148.849836:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233148.849846:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233148.849870:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233148.849897:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cb5be845 x1846780895117120/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:1.0:1761233149.617403:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233151.665474:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233152.689510:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233153.713451:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233153.969332:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233153.969344:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233153.969348:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233153.969359:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5714 00000100:00080000:2.0:1761233153.969362:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233153.969369:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233153.969419:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895117184/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233153.969455:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233153.969459:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233153.969460:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233153.969463:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5714 00000100:00080000:2.0:1761233153.969465:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233153.969469:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233153.969481:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233153.969496:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895117248/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233153.969511:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233153.969515:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233153.969516:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233153.969520:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5714 00000100:00080000:2.0:1761233153.969522:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233153.969525:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233153.969531:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233153.969544:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895117312/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233153.969552:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233153.969555:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233153.969556:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233153.969560:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5714 00000100:00080000:2.0:1761233153.969562:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233153.969565:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233153.969569:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233153.969580:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895117376/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233153.969588:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233153.969592:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233153.969593:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233153.969597:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5714 00000100:00080000:2.0:1761233153.969600:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233153.969605:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233153.969612:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233153.969624:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895117440/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233153.969646:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233153.969652:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233153.969654:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233153.969661:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5714 00000100:00080000:2.0:1761233153.969666:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233153.969671:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233153.969678:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233153.969690:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895117504/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233153.969698:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233153.969701:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233153.969703:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233153.969709:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5714 00000100:00080000:2.0:1761233153.969713:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233153.969718:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233153.969723:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233153.969734:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895117568/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:1.0:1761233155.761443:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233157.809531:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233159.089393:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233159.089406:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233159.089411:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233159.089423:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5720 00000100:00080000:2.0:1761233159.089427:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233159.089435:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233159.089491:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895117632/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233159.089514:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233159.089517:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233159.089518:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233159.089521:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5720 00000100:00080000:2.0:1761233159.089522:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233159.089525:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233159.089534:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233159.089542:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895117696/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233159.089548:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233159.089550:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233159.089551:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233159.089553:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5720 00000100:00080000:2.0:1761233159.089555:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233159.089557:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233159.089562:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233159.089567:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895117760/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233159.089573:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233159.089575:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233159.089575:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233159.089578:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5720 00000100:00080000:2.0:1761233159.089579:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233159.089581:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233159.089584:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233159.089592:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895117824/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233159.089596:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233159.089599:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233159.089600:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233159.089603:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5720 00000100:00080000:2.0:1761233159.089605:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233159.089607:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233159.089609:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233159.089615:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895117888/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233159.089621:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233159.089624:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233159.089625:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233159.089628:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5720 00000100:00080000:2.0:1761233159.089630:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233159.089633:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233159.089636:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233159.089641:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895117952/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233159.089646:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233159.089650:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233159.089651:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233159.089654:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5720 00000100:00080000:2.0:1761233159.089656:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233159.089658:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233159.089661:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233159.089670:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895118016/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:1.0:1761233159.857475:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233161.905446:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233163.953472:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233164.209438:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233164.209457:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233164.209464:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233164.209483:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5725 00000100:00080000:2.0:1761233164.209490:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233164.209503:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233164.209562:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895118080/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233164.209600:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233164.209609:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233164.209612:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233164.209621:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5725 00000100:00080000:2.0:1761233164.209626:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233164.209633:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233164.209650:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233164.209683:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895118144/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233164.209705:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233164.209712:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233164.209715:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233164.209723:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5725 00000100:00080000:2.0:1761233164.209728:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233164.209734:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233164.209744:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233164.209769:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895118208/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233164.209791:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233164.209798:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233164.209802:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233164.209809:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5725 00000100:00080000:2.0:1761233164.209818:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233164.209832:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233164.209848:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233164.209879:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895118272/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233164.209920:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233164.209931:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233164.209938:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233164.209952:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5725 00000100:00080000:2.0:1761233164.209959:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233164.209967:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233164.209977:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233164.210004:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895118336/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233164.210018:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233164.210043:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233164.210047:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233164.210053:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5725 00000100:00080000:2.0:1761233164.210056:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233164.210062:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233164.210076:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233164.210093:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895118400/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233164.210114:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233164.210119:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233164.210122:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233164.210128:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5725 00000100:00080000:2.0:1761233164.210132:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233164.210143:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233164.210156:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233164.210197:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895118464/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:1.0:1761233166.001385:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233168.049410:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233169.329365:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233169.329378:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233169.329384:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233169.329398:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5730 00000100:00080000:2.0:1761233169.329404:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233169.329412:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233169.329460:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895118528/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233169.329495:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233169.329499:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233169.329501:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233169.329506:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5730 00000100:00080000:2.0:1761233169.329508:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233169.329512:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233169.329523:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233169.329538:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895118592/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233169.329551:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233169.329558:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233169.329564:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233169.329575:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5730 00000100:00080000:2.0:1761233169.329578:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233169.329581:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233169.329587:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233169.329603:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895118656/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233169.329613:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233169.329617:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233169.329619:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233169.329624:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5730 00000100:00080000:2.0:1761233169.329626:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233169.329630:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233169.329634:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233169.329646:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895118720/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233169.329655:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233169.329664:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233169.329684:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233169.329692:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5730 00000100:00080000:2.0:1761233169.329698:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233169.329717:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233169.329726:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233169.329745:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895118784/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233169.329768:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233169.329778:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233169.329783:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233169.329792:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5730 00000100:00080000:2.0:1761233169.329795:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233169.329799:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233169.329804:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233169.329820:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895118848/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233169.329845:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233169.329850:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233169.329853:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233169.329858:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5730 00000100:00080000:2.0:1761233169.329860:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233169.329868:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233169.329876:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233169.329896:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895118912/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:1.0:1761233170.097496:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233172.145411:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233174.193618:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233174.449376:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233174.449390:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233174.449395:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233174.449411:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5735 00000100:00080000:2.0:1761233174.449416:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233174.449427:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233174.449472:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895118976/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233174.449508:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233174.449514:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233174.449516:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233174.449520:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5735 00000100:00080000:2.0:1761233174.449523:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233174.449526:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233174.449537:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233174.449554:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895119040/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233174.449566:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233174.449570:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233174.449572:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233174.449577:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5735 00000100:00080000:2.0:1761233174.449579:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233174.449588:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233174.449600:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233174.449615:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895119104/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233174.449627:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233174.449630:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233174.449632:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233174.449637:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5735 00000100:00080000:2.0:1761233174.449640:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233174.449643:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233174.449648:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233174.449660:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895119168/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233174.449669:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233174.449674:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233174.449675:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233174.449678:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5735 00000100:00080000:2.0:1761233174.449680:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233174.449689:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233174.449700:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233174.449717:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895119232/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233174.449734:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233174.449748:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233174.449751:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233174.449764:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5735 00000100:00080000:2.0:1761233174.449767:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233174.449775:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233174.449785:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233174.449804:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895119296/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233174.449848:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233174.449861:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233174.449864:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233174.449871:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5735 00000100:00080000:2.0:1761233174.449874:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233174.449878:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233174.449885:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233174.449902:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895119360/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:1.0:1761233176.241505:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233178.289452:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233179.569372:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233179.569385:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233179.569391:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233179.569405:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5740 00000100:00080000:2.0:1761233179.569410:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233179.569417:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233179.569460:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895119424/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233179.569494:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233179.569499:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233179.569501:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233179.569505:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5740 00000100:00080000:2.0:1761233179.569509:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233179.569514:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233179.569527:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233179.569549:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895119488/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233179.569571:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233179.569576:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233179.569579:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233179.569585:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5740 00000100:00080000:2.0:1761233179.569588:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233179.569592:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233179.569600:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233179.569617:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895119552/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233179.569628:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233179.569632:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233179.569634:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233179.569638:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5740 00000100:00080000:2.0:1761233179.569641:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233179.569650:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233179.569658:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233179.569679:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895119616/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233179.569755:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233179.569761:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233179.569766:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233179.569780:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5740 00000100:00080000:2.0:1761233179.569784:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233179.569791:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233179.569800:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233179.569820:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895119680/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233179.569836:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233179.569841:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233179.569845:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233179.569852:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5740 00000100:00080000:2.0:1761233179.569859:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233179.569868:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233179.569879:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233179.569899:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895119744/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233179.569935:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233179.569942:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233179.569944:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233179.569951:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5740 00000100:00080000:2.0:1761233179.569955:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233179.569962:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233179.569971:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233179.570016:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895119808/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:1.0:1761233180.337438:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233182.385334:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233184.433509:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233184.689328:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233184.689343:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233184.689348:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233184.689362:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5745 00000100:00080000:2.0:1761233184.689367:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233184.689375:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233184.689427:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895119872/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233184.689461:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233184.689465:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233184.689467:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233184.689472:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5745 00000100:00080000:2.0:1761233184.689475:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233184.689481:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233184.689492:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233184.689511:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895119936/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233184.689528:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233184.689534:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233184.689536:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233184.689542:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5745 00000100:00080000:2.0:1761233184.689545:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233184.689551:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233184.689560:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233184.689575:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895120000/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233184.689593:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233184.689599:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233184.689602:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233184.689612:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5745 00000100:00080000:2.0:1761233184.689617:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233184.689629:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233184.689638:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233184.689661:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895120064/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233184.689673:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233184.689678:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233184.689682:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233184.689691:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5745 00000100:00080000:2.0:1761233184.689696:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233184.689706:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233184.689715:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233184.689743:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895120128/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233184.689776:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233184.689787:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233184.689790:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233184.689803:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5745 00000100:00080000:2.0:1761233184.689809:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233184.689820:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233184.689834:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233184.689859:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895120192/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233184.689923:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233184.689928:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233184.689930:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233184.689936:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5745 00000100:00080000:2.0:1761233184.689938:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233184.689943:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233184.689949:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233184.689970:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895120256/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:1.0:1761233186.481481:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233188.529394:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233189.809461:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233189.809474:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233189.809479:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233189.809492:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5750 00000100:00080000:0.0:1761233189.809496:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233189.809505:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233189.809552:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895120320/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233189.809589:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233189.809593:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233189.809595:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233189.809601:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5750 00000100:00080000:0.0:1761233189.809604:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233189.809608:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233189.809621:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233189.809635:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895120384/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233189.809647:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233189.809654:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233189.809657:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233189.809668:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5750 00000100:00080000:0.0:1761233189.809672:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233189.809675:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233189.809680:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233189.809697:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895120448/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233189.809708:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233189.809712:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233189.809714:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233189.809719:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5750 00000100:00080000:0.0:1761233189.809722:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233189.809725:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233189.809729:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233189.809742:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895120512/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233189.809770:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233189.809776:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233189.809779:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233189.809786:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5750 00000100:00080000:0.0:1761233189.809789:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233189.809794:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233189.809803:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233189.809855:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895120576/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233189.809885:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233189.809893:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233189.809897:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233189.809910:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5750 00000100:00080000:0.0:1761233189.809914:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233189.809925:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233189.809936:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233189.809951:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895120640/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233189.809964:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233189.809972:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233189.809975:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233189.809984:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5750 00000100:00080000:0.0:1761233189.809987:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233189.809992:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233189.809998:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233189.810019:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895120704/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:1.0:1761233190.577516:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233192.625413:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233194.673532:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233194.929403:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233194.929416:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233194.929421:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233194.929434:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5755 00000100:00080000:2.0:1761233194.929439:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233194.929447:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233194.929492:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895120768/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233194.929532:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233194.929537:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233194.929539:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233194.929543:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5755 00000100:00080000:2.0:1761233194.929546:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233194.929549:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233194.929559:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233194.929598:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895120832/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233194.929609:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233194.929613:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233194.929615:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233194.929619:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5755 00000100:00080000:2.0:1761233194.929621:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233194.929624:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233194.929629:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233194.929638:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895120896/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233194.929647:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233194.929651:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233194.929653:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233194.929658:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5755 00000100:00080000:2.0:1761233194.929661:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233194.929665:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233194.929669:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233194.929683:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895120960/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233194.929695:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233194.929701:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233194.929706:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233194.929715:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5755 00000100:00080000:2.0:1761233194.929718:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233194.929722:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233194.929728:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233194.929800:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895121024/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233194.929811:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233194.929814:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233194.929816:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233194.929821:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5755 00000100:00080000:2.0:1761233194.929824:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233194.929828:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233194.929834:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233194.929852:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895121088/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233194.929863:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233194.929867:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233194.929869:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233194.929877:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5755 00000100:00080000:2.0:1761233194.929884:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233194.929892:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233194.929902:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233194.929917:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895121152/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:1.0:1761233196.721449:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233198.769465:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233200.049404:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233200.049414:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233200.049417:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233200.049428:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5760 00000100:00080000:0.0:1761233200.049432:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233200.049438:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233200.049474:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895121216/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233200.049499:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233200.049502:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233200.049503:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233200.049506:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5760 00000100:00080000:0.0:1761233200.049508:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233200.049510:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233200.049518:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233200.049529:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895121280/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233200.049537:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233200.049541:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233200.049543:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233200.049552:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5760 00000100:00080000:0.0:1761233200.049554:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233200.049556:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233200.049560:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233200.049569:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895121344/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233200.049576:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233200.049579:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233200.049581:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233200.049584:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5760 00000100:00080000:0.0:1761233200.049585:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233200.049587:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233200.049590:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233200.049597:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895121408/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233200.049603:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233200.049605:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233200.049606:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233200.049608:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5760 00000100:00080000:0.0:1761233200.049610:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233200.049616:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233200.049621:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233200.049630:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895121472/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233200.049647:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233200.049654:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233200.049656:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233200.049662:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5760 00000100:00080000:0.0:1761233200.049687:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233200.049695:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233200.049702:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233200.049714:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895121536/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233200.049731:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233200.049734:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233200.049736:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233200.049740:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5760 00000100:00080000:0.0:1761233200.049742:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233200.049744:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233200.049748:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233200.049757:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895121600/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:1.0:1761233200.817436:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233202.865417:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233204.913489:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233205.169374:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233205.169386:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233205.169390:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233205.169403:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5766 00000100:00080000:0.0:1761233205.169407:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233205.169416:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233205.169462:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895121664/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233205.169509:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233205.169513:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233205.169514:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233205.169518:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5766 00000100:00080000:0.0:1761233205.169520:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233205.169524:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233205.169536:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233205.169549:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895121728/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233205.169562:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233205.169568:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233205.169569:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233205.169573:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5766 00000100:00080000:0.0:1761233205.169575:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233205.169577:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233205.169581:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233205.169611:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895121792/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233205.169635:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233205.169638:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233205.169639:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233205.169642:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5766 00000100:00080000:0.0:1761233205.169644:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233205.169647:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233205.169650:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233205.169658:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895121856/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233205.169663:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233205.169667:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233205.169668:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233205.169671:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5766 00000100:00080000:0.0:1761233205.169672:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233205.169675:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233205.169678:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233205.169687:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895121920/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233205.169693:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233205.169696:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233205.169697:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233205.169700:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5766 00000100:00080000:0.0:1761233205.169702:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233205.169705:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233205.169708:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233205.169718:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895121984/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233205.169747:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233205.169760:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233205.169766:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233205.169770:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5766 00000100:00080000:0.0:1761233205.169776:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233205.169780:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233205.169785:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233205.169795:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895122048/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:1.0:1761233206.961342:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233209.009467:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233210.289279:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233210.289289:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233210.289292:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233210.289302:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5771 00000100:00080000:0.0:1761233210.289305:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233210.289312:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233210.289344:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895122112/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233210.289364:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233210.289367:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233210.289368:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233210.289370:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5771 00000100:00080000:0.0:1761233210.289372:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233210.289374:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233210.289383:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233210.289392:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895122176/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233210.289403:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233210.289406:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233210.289407:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233210.289410:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5771 00000100:00080000:0.0:1761233210.289412:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233210.289413:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233210.289417:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233210.289425:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895122240/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233210.289431:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233210.289433:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233210.289433:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233210.289436:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5771 00000100:00080000:0.0:1761233210.289437:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233210.289439:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233210.289442:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233210.289448:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895122304/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233210.289453:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233210.289455:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233210.289457:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233210.289459:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5771 00000100:00080000:0.0:1761233210.289461:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233210.289464:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233210.289466:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233210.289473:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895122368/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233210.289477:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233210.289480:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233210.289481:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233210.289487:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5771 00000100:00080000:0.0:1761233210.289490:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233210.289494:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233210.289498:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233210.289524:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895122432/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233210.289537:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233210.289541:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233210.289542:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233210.289546:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5771 00000100:00080000:0.0:1761233210.289548:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233210.289551:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233210.289558:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233210.289571:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895122496/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:1.0:1761233211.057465:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233213.105360:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1761233213.361541:0:2210:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:1.0:1761233213.361588:0:2210:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1761233213.361604:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233213.361649:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233213.361659:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233213.361667:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233213.361673:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233213.361680:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233213.361687:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00010000:00010000:1.0:1761233215.153391:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233215.409294:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233215.409303:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233215.409307:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233215.409316:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5776 00000100:00080000:0.0:1761233215.409319:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233215.409325:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233215.409353:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895122560/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233215.409374:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233215.409376:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233215.409377:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233215.409380:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5776 00000100:00080000:0.0:1761233215.409382:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233215.409385:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233215.409392:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233215.409400:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895122624/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233215.409408:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233215.409416:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233215.409418:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233215.409421:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5776 00000100:00080000:0.0:1761233215.409435:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233215.409438:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233215.409443:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233215.409452:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895122688/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233215.409458:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233215.409461:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233215.409462:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233215.409464:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5776 00000100:00080000:0.0:1761233215.409466:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233215.409468:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233215.409470:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233215.409476:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895122752/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233215.409481:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233215.409484:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233215.409485:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233215.409487:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5776 00000100:00080000:0.0:1761233215.409489:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233215.409496:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233215.409503:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233215.409516:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895122816/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233215.409525:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233215.409532:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233215.409535:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233215.409544:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5776 00000100:00080000:0.0:1761233215.409546:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233215.409553:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233215.409561:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233215.409575:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895122880/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233215.409593:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233215.409597:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233215.409599:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233215.409603:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5776 00000100:00080000:0.0:1761233215.409605:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233215.409609:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233215.409615:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233215.409631:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895122944/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00010000:00010000:1.0:1761233217.201376:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233218.225445:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233219.249440:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233220.273435:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233220.529329:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233220.529356:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233220.529360:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233220.529372:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5781 00000100:00080000:0.0:1761233220.529376:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233220.529383:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233220.529421:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000addc3db8 x1846780895123008/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233220.529449:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233220.529453:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233220.529455:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233220.529459:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5781 00000100:00080000:0.0:1761233220.529461:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233220.529466:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233220.529476:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233220.529481:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895123008:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761233220.529492:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000078e7052d x1846780895123072/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233220.529504:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233220.529507:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233220.529509:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233220.529513:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5781 00000100:00080000:0.0:1761233220.529516:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233220.529520:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233220.529525:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233220.529537:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895123136/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233220.529547:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1761233220.529547:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233220.529552:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233220.529555:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233220.529560:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5781 00000100:00080000:0.0:1761233220.529562:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233220.529567:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233220.529572:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233220.529572:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233220/real 1761233220] req@00000000addc3db8 x1846780895123008/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233245 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233220.529589:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895123200/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233220.529591:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000addc3db8 x1846780895123008/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233245 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233220.529617:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233220.529618:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233220.529622:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233220.529624:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1761233220.529627:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895123008:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233220.529628:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233220.529635:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5781 00000100:00080000:0.0:1761233220.529637:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233220.529641:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233220.529645:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233220.529648:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895123072:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233220.529658:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895123264/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233220.529668:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233220.529670:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233220.529675:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1761233220.529677:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233220/real 1761233220] req@0000000078e7052d x1846780895123072/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233245 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233220.529679:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233220.529688:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5781 00000100:00080000:0.0:1761233220.529691:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233220.529692:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000078e7052d x1846780895123072/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233245 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233220.529699:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:3.0:1761233220.529703:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1761233220.529707:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761233220.529707:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233220.529712:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895123072:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233220.529718:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895123136:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233220.529725:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895123328/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233220.529734:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233220.529737:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233220.529742:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233220/real 1761233220] req@000000005beaae31 x1846780895123136/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233245 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233220.529744:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233220.529746:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233220.529755:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5781 00000100:00080000:0.0:1761233220.529760:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233220.529766:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233220.529770:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895123136/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233245 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:0.0:1761233220.529772:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761233220.529780:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233220.529782:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233220.529784:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895123136:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233220.529788:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895123392/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233220.529791:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895123200:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233220.529810:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233220.529815:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233220/real 1761233220] req@0000000034d49f59 x1846780895123200/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233245 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233220.529822:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895123200/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233245 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233220.529828:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233220.529830:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233220.529834:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895123200:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233220.529850:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895123264:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233220.529862:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233220.529866:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233220/real 1761233220] req@00000000eca9ddc2 x1846780895123264/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233245 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233220.529872:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895123264/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233245 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233220.529876:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233220.529878:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233220.529880:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895123264:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233220.529883:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895123328:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233220.529892:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233220.529896:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233220/real 1761233220] req@000000006f2efc8b x1846780895123328/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233245 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233220.529901:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895123328/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233245 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233220.529905:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233220.529906:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233220.529908:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895123328:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233220.529911:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895123392:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233220.529919:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233220.529923:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233220/real 1761233220] req@000000004862b808 x1846780895123392/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233245 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233220.529928:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895123392/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233245 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233220.529932:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233220.529934:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233220.529935:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895123392:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233221.297328:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233222.321404:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233223.345380:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233224.369468:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233225.393412:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233225.649431:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233225.649442:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233225.649445:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233225.649456:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5786 00000100:00080000:2.0:1761233225.649459:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233225.649466:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233225.649505:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895123456/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233225.649535:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233225.649538:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233225.649540:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233225.649543:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5786 00000100:00080000:2.0:1761233225.649545:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233225.649548:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233225.649556:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233225.649567:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895123520/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233225.649577:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233225.649580:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233225.649583:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233225.649587:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5786 00000100:00080000:2.0:1761233225.649589:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233225.649591:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233225.649596:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233225.649599:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895123456:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761233225.649608:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895123584/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233225.649618:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233225.649621:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233225.649622:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233225.649626:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5786 00000100:00080000:2.0:1761233225.649628:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233225.649630:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233225.649634:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233225.649646:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895123648/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233225.649654:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233225.649656:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233225.649657:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233225.649660:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5786 00000100:00080000:2.0:1761233225.649662:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233225.649678:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233225.649682:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233225.649693:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233225.649694:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895123712/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233225.649704:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233225.649708:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233225.649710:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233225.649715:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5786 00000100:00080000:2.0:1761233225.649716:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233225.649720:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233225.649723:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233225.649724:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233225/real 1761233225] req@000000004f29ca26 x1846780895123456/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233250 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233225.649735:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895123776/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233225.649748:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233225.649751:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233225.649753:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233225.649755:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895123456/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233250 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233225.649760:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5786 00000100:00080000:2.0:1761233225.649766:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233225.649772:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:3.0:1761233225.649777:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1761233225.649778:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761233225.649786:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233225.649790:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895123840/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233225.649799:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895123456:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233225.649815:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895123520:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233225.649849:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233225.649860:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233225/real 1761233225] req@0000000033765045 x1846780895123520/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233250 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233225.649877:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895123520/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233250 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233225.649903:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233225.649906:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233225.649911:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895123520:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233225.649955:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895123584:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233225.649985:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233225.649997:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233225/real 1761233225] req@000000009d315fa5 x1846780895123584/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233250 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233225.650022:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895123584/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233250 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233225.650033:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233225.650036:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233225.650041:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895123584:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233225.650050:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895123648:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233225.650081:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233225.650091:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233225/real 1761233225] req@0000000040d5f361 x1846780895123648/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233250 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233225.650102:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895123648/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233250 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233225.650109:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233225.650111:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233225.650114:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895123648:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233225.650119:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895123712:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233225.650136:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233225.650142:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233225/real 1761233225] req@000000007dcf84ae x1846780895123712/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233250 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233225.650151:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895123712/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233250 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233225.650158:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233225.650161:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233225.650164:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895123712:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233225.650168:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895123776:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233225.650183:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233225.650188:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233225/real 1761233225] req@000000007091f547 x1846780895123776/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233250 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233225.650197:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895123776/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233250 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233225.650204:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233225.650206:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233225.650208:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895123776:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233225.650212:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895123840:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233225.650294:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233225.650303:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233225/real 1761233225] req@0000000064a2cea0 x1846780895123840/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233250 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233225.650312:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895123840/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233250 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233225.650319:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233225.650320:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233225.650323:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895123840:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233226.417461:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233227.441404:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233228.465420:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233229.489415:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233230.513427:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233230.769390:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233230.769404:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233230.769411:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233230.769429:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5791 00000100:00080000:2.0:1761233230.769434:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233230.769444:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233230.769489:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895123904/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233230.769524:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233230.769528:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233230.769530:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233230.769535:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5791 00000100:00080000:2.0:1761233230.769538:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233230.769542:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233230.769553:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233230.769570:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895123968/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233230.769595:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895123904:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761233230.769604:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233230.769615:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233230.769617:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233230.769623:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5791 00000100:00080000:2.0:1761233230.769627:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233230.769632:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233230.769640:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233230.769660:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895124032/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233230.769671:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233230.769675:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233230.769678:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233230.769683:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5791 00000100:00080000:2.0:1761233230.769686:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233230.769691:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233230.769696:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233230.769707:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895124096/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233230.769716:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233230.769720:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233230.769723:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1761233230.769724:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233230.769737:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5791 00000100:00080000:2.0:1761233230.769740:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233230.769748:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233230.769755:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233230.769779:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233230/real 1761233230] req@0000000064a2cea0 x1846780895123904/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233255 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233230.769780:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895124160/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233230.769845:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233230.769851:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233230.769856:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233230.769867:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5791 00000100:00100000:3.0:1761233230.769869:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895123904/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233255 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233230.769873:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233230.769887:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233230.769902:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761233230.769910:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233230.769918:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233230.769919:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895124224/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233230.769942:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895123904:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761233230.769969:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233230.769974:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233230.769977:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233230.769982:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5791 00000100:00080000:2.0:1761233230.769984:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233230.769990:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233230.769996:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233230.769999:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895123968:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233230.770014:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895124288/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233230.770039:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233230.770054:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233230/real 1761233230] req@000000007091f547 x1846780895123968/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233255 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233230.770083:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895123968/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233255 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233230.770091:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233230.770094:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233230.770097:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895123968:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233230.770104:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895124032:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233230.770125:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233230.770132:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233230/real 1761233230] req@000000007dcf84ae x1846780895124032/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233255 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233230.770148:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895124032/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233255 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233230.770155:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233230.770157:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233230.770159:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895124032:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233230.770164:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895124096:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233230.770238:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233230.770248:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233230/real 1761233230] req@0000000040d5f361 x1846780895124096/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233255 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233230.770259:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895124096/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233255 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233230.770266:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233230.770269:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233230.770271:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895124096:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233230.770276:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895124160:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233230.770296:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233230.770302:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233230/real 1761233230] req@000000009d315fa5 x1846780895124160/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233255 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233230.770312:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895124160/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233255 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233230.770318:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233230.770322:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233230.770324:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895124160:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233230.770330:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895124224:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233230.770345:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233230.770351:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233230/real 1761233230] req@0000000033765045 x1846780895124224/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233255 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233230.770359:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895124224/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233255 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233230.770366:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233230.770368:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233230.770370:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895124224:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233230.770395:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895124288:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233230.770411:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233230.770416:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233230/real 1761233230] req@000000004f29ca26 x1846780895124288/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233255 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233230.770425:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895124288/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233255 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233230.770432:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233230.770434:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233230.770437:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895124288:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233231.537484:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233232.561465:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233233.585480:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233234.609554:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233235.633448:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233235.889426:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233235.889439:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233235.889444:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233235.889457:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5796 00000100:00080000:0.0:1761233235.889461:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233235.889468:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233235.889510:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895124352/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233235.889540:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233235.889543:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233235.889545:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233235.889549:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5796 00000100:00080000:0.0:1761233235.889551:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233235.889554:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233235.889563:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233235.889592:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895124416/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233235.889611:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895124352:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233235.889621:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233235.889628:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233235.889631:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233235.889639:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5796 00000100:00080000:0.0:1761233235.889644:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233235.889650:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233235.889658:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233235.889680:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895124480/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233235.889690:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233235.889693:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233235.889696:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233235.889700:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5796 00000100:00080000:0.0:1761233235.889702:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233235.889705:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233235.889711:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233235.889714:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233235.889726:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895124544/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233235.889736:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233235.889740:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233235.889742:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233235.889747:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5796 00000100:00080000:0.0:1761233235.889749:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233235.889754:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233235/real 1761233235] req@00000000c6e06344 x1846780895124352/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233260 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233235.889756:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233235.889763:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233235.889775:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895124608/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233235.889787:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233235.889791:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233235.889794:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233235.889795:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895124352/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233260 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233235.889802:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5796 00000100:00080000:0.0:1761233235.889808:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233235.889814:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233235.889822:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761233235.889834:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1761233235.889838:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895124672/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233235.889843:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233235.889854:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233235.889856:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895124352:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233235.889860:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233235.889863:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233235.889873:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5796 00000100:00080000:0.0:1761233235.889875:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233235.889879:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233235.889883:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233235.889893:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895124736/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233235.889907:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895124416:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233235.889949:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233235.889964:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233235/real 1761233235] req@000000005593c994 x1846780895124416/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233260 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233235.889982:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895124416/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233260 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233235.889995:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233235.889998:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233235.890004:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895124416:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233235.890013:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895124480:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233235.890043:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233235.890055:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233235/real 1761233235] req@000000001f9ec359 x1846780895124480/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233260 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233235.890081:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895124480/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233260 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233235.890093:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233235.890097:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233235.890158:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895124480:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233235.890169:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895124544:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233235.890241:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233235.890259:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233235/real 1761233235] req@00000000efa29eb9 x1846780895124544/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233260 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233235.890276:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895124544/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233260 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233235.890288:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233235.890291:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233235.890296:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895124544:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233235.890305:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895124608:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233235.890337:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233235.890350:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233235/real 1761233235] req@000000006d61dd06 x1846780895124608/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233260 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233235.890367:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895124608/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233260 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233235.890378:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233235.890382:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233235.890387:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895124608:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233235.890395:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895124672:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233235.890426:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233235.890439:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233235/real 1761233235] req@00000000ef655201 x1846780895124672/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233260 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233235.890456:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895124672/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233260 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233235.890467:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233235.890470:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233235.890474:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895124672:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233235.890510:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895124736:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233235.890542:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233235.890555:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233235/real 1761233235] req@000000004130b6bd x1846780895124736/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233260 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233235.890572:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895124736/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233260 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233235.890583:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233235.890587:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233235.890591:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895124736:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233236.657444:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233237.681426:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233238.705490:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233239.729684:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233240.753411:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233241.009323:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233241.009334:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233241.009340:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233241.009353:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5801 00000100:00080000:0.0:1761233241.009357:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233241.009366:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233241.009411:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895124800/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233241.009440:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233241.009443:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233241.009444:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233241.009446:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5801 00000100:00080000:0.0:1761233241.009448:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233241.009451:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233241.009460:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233241.009468:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895124864/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233241.009474:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233241.009476:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233241.009477:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233241.009480:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5801 00000100:00080000:0.0:1761233241.009481:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233241.009482:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233241.009485:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233241.009490:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895124928/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233241.009495:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233241.009496:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233241.009497:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233241.009501:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5801 00000100:00080000:0.0:1761233241.009503:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233241.009505:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233241.009508:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233241.009514:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895124992/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233241.009519:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233241.009521:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233241.009522:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233241.009524:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5801 00000100:00080000:0.0:1761233241.009525:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233241.009527:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233241.009530:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233241.009536:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895124800:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233241.009538:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895125056/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233241.009549:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233241.009550:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233241.009551:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233241.009553:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5801 00000100:00080000:0.0:1761233241.009554:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233241.009557:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233241.009559:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233241.009567:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895125120/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233241.009576:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233241.009578:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233241.009580:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233241.009582:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5801 00000100:00080000:0.0:1761233241.009583:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233241.009586:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233241.009588:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233241.009595:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895125184/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233241.009625:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233241.009650:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233241/real 1761233241] req@000000007bd1ec64 x1846780895124800/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233266 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233241.009664:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895124800/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233266 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233241.009672:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233241.009676:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233241.009679:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895124800:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233241.009684:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895124864:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233241.009698:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233241.009703:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233241/real 1761233241] req@000000001db96463 x1846780895124864/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233266 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233241.009709:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895124864/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233266 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233241.009714:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233241.009715:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233241.009717:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895124864:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233241.009720:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895124928:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233241.009731:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233241.009735:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233241/real 1761233241] req@000000009d1c7f73 x1846780895124928/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233266 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233241.009746:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895124928/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233266 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233241.009750:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233241.009751:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233241.009753:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895124928:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233241.009756:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895124992:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233241.009766:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233241.009769:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233241/real 1761233241] req@00000000d67bb589 x1846780895124992/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233266 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233241.009775:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895124992/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233266 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233241.009779:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233241.009780:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233241.009781:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895124992:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233241.009807:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895125056:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233241.009818:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233241.009822:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233241/real 1761233241] req@00000000a4a2c44a x1846780895125056/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233266 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233241.009827:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895125056/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233266 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233241.009831:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233241.009834:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233241.009835:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895125056:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233241.009839:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895125120:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233241.009849:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233241.009853:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233241/real 1761233241] req@000000004a74da19 x1846780895125120/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233266 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233241.009858:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895125120/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233266 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233241.009863:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233241.009865:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233241.009866:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895125120:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233241.009869:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895125184:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233241.009878:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233241.009881:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233241/real 1761233241] req@000000002d1b8ef7 x1846780895125184/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233266 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233241.009887:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895125184/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233266 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233241.009891:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233241.009892:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233241.009893:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895125184:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233241.777396:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233242.801358:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233243.825529:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233244.849412:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233245.873469:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233246.129348:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233246.129359:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233246.129363:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233246.129374:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5807 00000100:00080000:0.0:1761233246.129378:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233246.129384:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233246.129419:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895125248/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233246.129447:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233246.129452:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233246.129453:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233246.129457:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5807 00000100:00080000:0.0:1761233246.129460:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233246.129464:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233246.129478:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233246.129493:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895125312/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233246.129503:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233246.129507:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233246.129509:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233246.129513:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5807 00000100:00080000:0.0:1761233246.129515:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233246.129518:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233246.129521:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895125248:192.168.201.157@tcp:250:kworker/u8:2.0 00000008:00100000:0.0:1761233246.129523:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233246.129535:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895125376/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233246.129544:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233246.129547:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233246.129548:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233246.129552:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5807 00000100:00080000:0.0:1761233246.129554:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233246.129558:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233246.129562:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233246.129570:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895125440/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233246.129578:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233246.129581:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233246.129583:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233246.129587:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5807 00000100:00080000:0.0:1761233246.129589:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233246.129593:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:3.0:1761233246.129593:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:0.0:1761233246.129597:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233246.129605:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895125504/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233246.129613:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233246.129617:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233246.129618:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233246.129618:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233246/real 1761233246] req@000000005d83493f x1846780895125248/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233271 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233246.129622:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5807 00000100:00080000:0.0:1761233246.129625:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233246.129628:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233246.129632:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233246.129637:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005d83493f x1846780895125248/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233271 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233246.129641:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895125568/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233246.129651:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233246.129651:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233246.129654:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233246.129654:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233246.129656:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233246.129659:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5807 00000100:00100000:3.0:1761233246.129659:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895125248:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233246.129662:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233246.129665:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233246.129667:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895125312:192.168.201.157@tcp:38:kworker/u8:2.0 00000008:00100000:0.0:1761233246.129668:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233246.129677:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895125632/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233246.129691:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233246.129701:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233246/real 1761233246] req@00000000f3ed2588 x1846780895125312/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233271 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233246.129714:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f3ed2588 x1846780895125312/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233271 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233246.129723:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233246.129726:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233246.129729:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895125312:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233246.129758:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895125376:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233246.129778:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233246.129785:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233246/real 1761233246] req@000000002d1b8ef7 x1846780895125376/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233271 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233246.129798:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895125376/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233271 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233246.129804:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233246.129806:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233246.129808:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895125376:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233246.129812:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895125440:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233246.129823:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233246.129828:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233246/real 1761233246] req@000000004a74da19 x1846780895125440/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233271 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233246.129836:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895125440/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233271 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233246.129841:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233246.129843:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233246.129844:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895125440:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233246.129848:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895125504:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233246.129860:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233246.129864:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233246/real 1761233246] req@00000000a4a2c44a x1846780895125504/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233271 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233246.129872:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895125504/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233271 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233246.129877:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233246.129879:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233246.129881:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895125504:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233246.129885:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895125568:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233246.129896:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233246.129901:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233246/real 1761233246] req@00000000d67bb589 x1846780895125568/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233271 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233246.129908:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895125568/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233271 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233246.129914:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233246.129915:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233246.129917:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895125568:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233246.129920:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895125632:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233246.129931:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233246.129936:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233246/real 1761233246] req@000000009d1c7f73 x1846780895125632/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233271 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233246.129960:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895125632/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233271 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233246.129966:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233246.129968:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233246.129986:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895125632:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233246.897532:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233247.921428:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233248.945440:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233249.969401:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233250.993450:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233251.249280:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233251.249289:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233251.249293:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233251.249302:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5812 00000100:00080000:0.0:1761233251.249305:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233251.249311:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233251.249342:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895125696/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233251.249366:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233251.249369:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233251.249370:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233251.249372:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5812 00000100:00080000:0.0:1761233251.249374:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233251.249376:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233251.249382:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233251.249389:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895125760/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233251.249395:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233251.249397:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233251.249398:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233251.249400:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5812 00000100:00080000:0.0:1761233251.249401:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233251.249403:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233251.249408:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233251.249414:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895125824/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233251.249420:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233251.249422:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233251.249422:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233251.249425:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5812 00000100:00080000:0.0:1761233251.249426:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233251.249428:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233251.249431:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233251.249439:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895125888/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233251.249440:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895125696:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233251.249452:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233251.249454:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233251.249454:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233251.249457:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5812 00000100:00080000:0.0:1761233251.249458:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233251.249460:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233251.249463:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233251.249470:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895125952/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233251.249474:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233251.249476:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233251.249477:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233251.249480:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5812 00000100:00080000:0.0:1761233251.249481:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233251.249483:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233251.249486:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233251.249492:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895126016/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233251.249496:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233251.249498:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233251.249499:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233251.249501:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5812 00000100:00080000:0.0:1761233251.249502:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233251.249505:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233251.249507:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233251.249513:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895126080/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233251.249521:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233251.249549:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233251/real 1761233251] req@000000001db96463 x1846780895125696/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233276 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233251.249567:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895125696/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233276 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233251.249578:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233251.249581:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233251.249585:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895125696:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233251.249593:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895125760:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233251.249618:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233251.249628:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233251/real 1761233251] req@000000007bd1ec64 x1846780895125760/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233276 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233251.249640:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895125760/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233276 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233251.249649:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233251.249651:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233251.249654:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895125760:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233251.249660:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895125824:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233251.249679:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233251.249687:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233251/real 1761233251] req@000000004130b6bd x1846780895125824/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233276 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233251.249705:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895125824/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233276 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233251.249713:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233251.249715:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233251.249718:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895125824:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233251.249744:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895125888:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233251.249778:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233251.249786:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233251/real 1761233251] req@00000000ef655201 x1846780895125888/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233276 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233251.249797:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895125888/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233276 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233251.249811:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233251.249814:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233251.249817:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895125888:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233251.249823:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895125952:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233251.249844:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233251.249853:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233251/real 1761233251] req@000000006d61dd06 x1846780895125952/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233276 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233251.249879:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895125952/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233276 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233251.249886:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233251.249889:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233251.249892:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895125952:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233251.249898:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895126016:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233251.249918:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233251.249926:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233251/real 1761233251] req@00000000efa29eb9 x1846780895126016/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233276 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233251.249937:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895126016/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233276 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233251.249944:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233251.249947:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233251.249949:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895126016:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233251.249954:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895126080:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233251.249971:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233251.249978:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233251/real 1761233251] req@000000001f9ec359 x1846780895126080/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233276 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233251.249988:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895126080/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233276 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233251.249996:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233251.249998:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233251.250000:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895126080:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233252.017467:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233253.041540:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233254.065362:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233255.089409:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233256.113422:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233256.369346:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233256.369360:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233256.369365:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233256.369380:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5817 00000100:00080000:0.0:1761233256.369384:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233256.369393:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233256.369438:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895126144/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233256.369475:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233256.369480:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233256.369482:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233256.369486:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5817 00000100:00080000:0.0:1761233256.369488:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233256.369492:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233256.369503:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233256.369517:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895126208/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233256.369528:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895126144:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233256.369529:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233256.369537:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233256.369540:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233256.369545:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5817 00000100:00080000:0.0:1761233256.369547:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233256.369551:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233256.369557:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233256.369568:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895126272/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233256.369577:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233256.369580:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233256.369581:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233256.369584:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5817 00000100:00080000:0.0:1761233256.369586:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233256.369589:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233256.369593:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233256.369602:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895126336/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233256.369625:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233256.369628:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233256.369631:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1761233256.369635:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233256.369639:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5817 00000100:00080000:0.0:1761233256.369643:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233256.369648:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233256.369653:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233256.369666:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895126400/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233256.369671:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233256/real 1761233256] req@000000005593c994 x1846780895126144/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233281 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233256.369691:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233256.369697:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233256.369699:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233256.369702:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895126144/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233281 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233256.369708:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5817 00000100:00080000:0.0:1761233256.369714:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233256.369719:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233256.369721:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:3.0:1761233256.369723:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000008:00100000:0.0:1761233256.369730:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233256.369731:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895126144:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233256.369747:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895126464/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233256.369760:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233256.369763:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895126208:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:0.0:1761233256.369770:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233256.369773:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233256.369794:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5817 00000100:00080000:0.0:1761233256.369798:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233256.369803:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233256.369809:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233256.369816:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233256.369827:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895126528/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233256.369834:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233256/real 1761233256] req@00000000c6e06344 x1846780895126208/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233281 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233256.369854:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895126208/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233281 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233256.369864:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233256.369867:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233256.369871:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895126208:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233256.369878:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895126272:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233256.369902:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233256.369909:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233256/real 1761233256] req@000000004862b808 x1846780895126272/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233281 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233256.369923:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895126272/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233281 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233256.369929:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233256.369930:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233256.369933:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895126272:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233256.369937:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895126336:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233256.369948:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233256.369953:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233256/real 1761233256] req@000000006f2efc8b x1846780895126336/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233281 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233256.369959:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895126336/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233281 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233256.369965:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233256.369966:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233256.369968:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895126336:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233256.369972:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895126400:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233256.369991:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233256.369999:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233256/real 1761233256] req@00000000eca9ddc2 x1846780895126400/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233281 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233256.370012:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895126400/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233281 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233256.370019:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233256.370023:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233256.370026:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895126400:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233256.370048:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895126464:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233256.370072:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233256.370079:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233256/real 1761233256] req@0000000034d49f59 x1846780895126464/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233281 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233256.370087:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895126464/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233281 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233256.370093:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233256.370094:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233256.370096:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895126464:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233256.370100:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895126528:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233256.370112:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233256.370117:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233256/real 1761233256] req@000000005beaae31 x1846780895126528/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233281 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233256.370124:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895126528/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233281 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233256.370129:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233256.370131:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233256.370132:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895126528:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233257.137419:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233258.161490:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233259.185469:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233260.209447:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233261.233580:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233261.489453:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233261.489469:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233261.489475:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233261.489491:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5822 00000100:00080000:2.0:1761233261.489495:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233261.489505:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233261.489555:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895126592/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233261.489588:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233261.489595:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233261.489598:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233261.489604:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5822 00000100:00080000:2.0:1761233261.489608:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233261.489614:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233261.489632:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233261.489653:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895126656/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233261.489671:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233261.489677:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233261.489679:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233261.489685:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5822 00000100:00080000:2.0:1761233261.489689:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233261.489694:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233261.489695:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895126592:192.168.201.157@tcp:250:kworker/u8:2.0 00000008:00100000:2.0:1761233261.489727:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233261.489747:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895126720/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233261.489763:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233261.489768:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233261.489770:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233261.489776:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5822 00000100:00080000:2.0:1761233261.489779:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233261.489784:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233261.489790:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233261.489804:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895126784/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233261.489818:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1761233261.489820:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233261.489824:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233261.489827:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233261.489833:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5822 00000100:00080000:2.0:1761233261.489836:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233261.489842:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233261.489849:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233261.489854:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233261/real 1761233261] req@00000000fd3935a6 x1846780895126592/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233286 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233261.489864:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895126848/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233261.489879:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233261.489883:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895126592/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233286 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233261.489885:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233261.489887:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233261.489894:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5822 00000100:00080000:2.0:1761233261.489897:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233261.489901:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233261.489903:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:3.0:1761233261.489906:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000008:00100000:2.0:1761233261.489909:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233261.489912:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895126592:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761233261.489923:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895126912/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233261.489924:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895126656:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:2.0:1761233261.489938:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233261.489943:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233261.489946:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233261.489952:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5822 00000100:00080000:2.0:1761233261.489955:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233261.489962:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:3.0:1761233261.489963:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:2.0:1761233261.489969:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233261.489978:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233261/real 1761233261] req@00000000135e9a6e x1846780895126656/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233286 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233261.489984:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895126976/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233261.490000:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895126656/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233286 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233261.490015:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233261.490019:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233261.490024:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895126656:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233261.490067:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895126720:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233261.490108:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233261.490120:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233261/real 1761233261] req@000000006ff1fdbc x1846780895126720/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233286 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233261.490142:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895126720/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233286 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233261.490151:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233261.490154:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233261.490159:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895126720:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233261.490169:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895126784:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233261.490254:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233261.490274:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233261/real 1761233261] req@0000000088939481 x1846780895126784/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233286 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233261.490289:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895126784/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233286 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233261.490298:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233261.490301:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233261.490304:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895126784:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233261.490311:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895126848:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233261.490340:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233261.490349:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233261/real 1761233261] req@000000006d8c3eb0 x1846780895126848/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233286 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233261.490361:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895126848/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233286 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233261.490369:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233261.490373:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233261.490376:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895126848:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233261.490383:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895126912:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233261.490400:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233261.490408:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233261/real 1761233261] req@000000006c1041ba x1846780895126912/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233286 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233261.490419:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895126912/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233286 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233261.490427:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233261.490430:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233261.490433:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895126912:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233261.490438:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895126976:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233261.490455:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233261.490463:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233261/real 1761233261] req@00000000fd4aa3a2 x1846780895126976/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233286 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233261.490473:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895126976/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233286 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233261.490482:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233261.490484:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233261.490487:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895126976:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233262.257476:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233263.281427:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233264.305362:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233265.329491:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233266.353598:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233266.609370:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233266.609384:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233266.609389:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233266.609404:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5827 00000100:00080000:0.0:1761233266.609408:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233266.609417:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233266.609463:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000078e7052d x1846780895127040/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233266.609503:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233266.609509:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233266.609511:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233266.609516:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5827 00000100:00080000:0.0:1761233266.609519:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233266.609524:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233266.609536:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233266.609553:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000addc3db8 x1846780895127104/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233266.609566:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233266.609570:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233266.609573:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233266.609578:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5827 00000100:00080000:0.0:1761233266.609581:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233266.609588:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233266.609595:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233266.609614:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895127168/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233266.609614:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895127040:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233266.609653:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233266.609658:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233266.609660:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233266.609667:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5827 00000100:00080000:0.0:1761233266.609671:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233266.609676:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233266.609683:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233266.609701:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895127232/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233266.609711:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233266.609715:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233266.609717:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233266.609721:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5827 00000100:00080000:0.0:1761233266.609724:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233266.609730:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233266.609736:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233266.609739:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233266.609755:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895127296/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233266.609767:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233266/real 1761233266] req@0000000078e7052d x1846780895127040/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233291 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233266.609769:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233266.609781:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233266.609786:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233266.609795:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5827 00000100:00100000:3.0:1761233266.609796:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000078e7052d x1846780895127040/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233291 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233266.609803:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233266.609814:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233266.609815:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:3.0:1761233266.609820:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000008:00100000:0.0:1761233266.609825:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233266.609828:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895127040:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233266.609839:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895127104:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233266.609847:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895127360/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233266.609861:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1761233266.609862:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233266.609869:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233266.609871:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233266.609873:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233266/real 1761233266] req@00000000addc3db8 x1846780895127104/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233291 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233266.609882:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5827 00000100:00080000:0.0:1761233266.609889:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233266.609894:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000addc3db8 x1846780895127104/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233291 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233266.609898:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233266.609910:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761233266.609911:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233266.609918:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233266.609922:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895127104:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233266.609935:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895127424/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233266.609952:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895127168:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233266.609972:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233266.609980:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233266/real 1761233266] req@000000005beaae31 x1846780895127168/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233291 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233266.609995:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895127168/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233291 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233266.610001:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233266.610003:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233266.610005:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895127168:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233266.610010:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895127232:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233266.610022:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233266.610028:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233266/real 1761233266] req@0000000034d49f59 x1846780895127232/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233291 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233266.610036:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895127232/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233291 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233266.610042:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233266.610044:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233266.610046:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895127232:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233266.610050:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895127296:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233266.610063:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233266.610068:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233266/real 1761233266] req@00000000eca9ddc2 x1846780895127296/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233291 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233266.610076:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895127296/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233291 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233266.610082:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233266.610085:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233266.610087:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895127296:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233266.610092:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895127360:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233266.610104:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233266.610109:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233266/real 1761233266] req@000000006f2efc8b x1846780895127360/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233291 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233266.610117:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895127360/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233291 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233266.610122:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233266.610124:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233266.610126:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895127360:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233266.610140:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895127424:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233266.610152:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233266.610158:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233266/real 1761233266] req@000000004862b808 x1846780895127424/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233291 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233266.610165:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895127424/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233291 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233266.610170:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233266.610172:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233266.610174:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895127424:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233267.377436:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233268.401415:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233269.425442:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233270.449362:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233271.473495:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233271.729356:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233271.729365:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233271.729369:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233271.729379:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5832 00000100:00080000:2.0:1761233271.729382:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233271.729389:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233271.729424:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895127488/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233271.729445:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233271.729447:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233271.729448:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233271.729451:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5832 00000100:00080000:2.0:1761233271.729453:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233271.729455:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233271.729462:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233271.729470:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895127552/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233271.729476:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233271.729478:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233271.729479:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233271.729482:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5832 00000100:00080000:2.0:1761233271.729483:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233271.729485:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233271.729490:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233271.729495:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895127616/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233271.729501:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233271.729503:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233271.729505:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233271.729507:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5832 00000100:00080000:2.0:1761233271.729509:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233271.729511:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233271.729513:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233271.729521:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895127680/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233271.729528:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895127488:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761233271.729529:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233271.729535:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233271.729550:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233271.729555:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5832 00000100:00080000:2.0:1761233271.729557:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233271.729560:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233271.729563:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233271.729572:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895127744/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233271.729578:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233271.729581:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233271.729582:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233271.729585:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5832 00000100:00080000:2.0:1761233271.729586:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233271.729589:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233271.729592:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233271.729599:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895127808/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233271.729603:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233271.729605:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233271.729606:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233271.729609:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5832 00000100:00080000:2.0:1761233271.729611:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233271.729613:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233271.729616:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233271.729625:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895127872/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233271.729625:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233271.729655:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233271/real 1761233271] req@00000000c10cebe6 x1846780895127488/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233296 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233271.729674:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895127488/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233296 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233271.729685:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233271.729688:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233271.729692:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895127488:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233271.729701:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895127552:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233271.729727:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233271.729737:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233271/real 1761233271] req@00000000a1e8c6bb x1846780895127552/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233296 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233271.729750:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895127552/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233296 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233271.729757:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233271.729760:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233271.729764:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895127552:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233271.729770:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895127616:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233271.729794:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233271.729804:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233271/real 1761233271] req@000000008ebccab5 x1846780895127616/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233296 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233271.729823:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895127616/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233296 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233271.729831:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233271.729833:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233271.729837:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895127616:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233271.729866:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895127680:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233271.729888:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233271.729899:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233271/real 1761233271] req@000000004ae5bc9b x1846780895127680/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233296 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233271.729910:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895127680/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233296 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233271.729918:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233271.729920:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233271.729923:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895127680:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233271.729930:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895127744:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233271.729950:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233271.729959:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233271/real 1761233271] req@00000000ac7d8d5b x1846780895127744/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233296 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233271.729971:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895127744/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233296 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233271.729979:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233271.729983:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233271.729986:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895127744:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233271.729992:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895127808:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233271.730014:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233271.730023:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233271/real 1761233271] req@00000000f4e415d3 x1846780895127808/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233296 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233271.730034:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895127808/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233296 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233271.730042:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233271.730044:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233271.730047:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895127808:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233271.730053:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895127872:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233271.730075:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233271.730083:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233271/real 1761233271] req@0000000099b7e5d9 x1846780895127872/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233296 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233271.730095:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895127872/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233296 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233271.730102:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233271.730105:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233271.730108:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895127872:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233272.497417:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233273.521412:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233274.545517:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233275.569444:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233276.593431:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233276.849322:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233276.849331:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233276.849334:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233276.849345:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5837 00000100:00080000:2.0:1761233276.849348:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233276.849353:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233276.849387:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895127936/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233276.849414:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233276.849418:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233276.849419:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233276.849422:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5837 00000100:00080000:2.0:1761233276.849424:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233276.849427:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233276.849434:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233276.849448:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895128000/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233276.849470:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233276.849473:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233276.849475:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233276.849478:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5837 00000100:00080000:2.0:1761233276.849479:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233276.849482:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233276.849486:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233276.849494:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895127936:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761233276.849498:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895128064/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233276.849518:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233276.849520:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233276.849521:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233276.849524:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5837 00000100:00080000:2.0:1761233276.849525:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233276.849527:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233276.849529:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233276.849535:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895128128/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233276.849540:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233276.849543:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233276.849544:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233276.849546:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5837 00000100:00080000:2.0:1761233276.849547:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233276.849549:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233276.849551:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233276.849560:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895128192/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233276.849565:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233276.849567:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233276.849568:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233276.849570:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5837 00000100:00080000:2.0:1761233276.849571:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233276.849574:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233276.849576:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233276.849581:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895128256/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233276.849586:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233276.849588:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233276.849589:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233276.849593:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5837 00000100:00080000:2.0:1761233276.849595:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233276.849597:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233276.849599:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233276.849607:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895128320/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233276.849628:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895128000:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761233276.849677:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895128064:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233276.849713:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895128128:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233276.849741:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895128192:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761233276.849770:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895128256:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233276.849796:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895128320:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233277.617414:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233278.641487:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233279.665357:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1761233279.921661:0:2207:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:0.0:1761233279.921714:0:2207:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1761233279.921733:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233279.921785:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233279.921796:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233279.921808:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:3.0:1761233279.921817:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233276/real 1761233279] req@00000000ac7d8d5b x1846780895128064/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233301 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000800:00000100:0.0:1761233279.921823:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233279.921847:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:3.0:1761233279.921853:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895128064/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233301 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000800:00000100:0.0:1761233279.921864:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00080000:3.0:1761233279.921872:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233279.921878:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233279.921885:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895128064:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233279.921893:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233276/real 1761233279] req@000000004ae5bc9b x1846780895128128/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233301 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233279.921907:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895128128/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233301 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233279.921914:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233279.921918:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233279.921921:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895128128:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233279.921927:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233276/real 1761233279] req@000000008ebccab5 x1846780895128192/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233301 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233279.921949:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895128192/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233301 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233279.921958:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233279.921961:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233279.921965:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895128192:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233279.921973:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233276/real 1761233279] req@00000000a1e8c6bb x1846780895128256/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233301 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233279.921985:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895128256/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233301 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233279.921993:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233279.921996:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233279.921999:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895128256:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233279.922005:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233276/real 1761233279] req@00000000c10cebe6 x1846780895128320/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233301 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233279.922015:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895128320/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233301 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233279.922023:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233279.922026:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233279.922029:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895128320:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233279.922035:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233276/real 1761233279] req@0000000099b7e5d9 x1846780895127936/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233301 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233279.922048:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895127936/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233301 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233279.922056:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233279.922059:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233279.922062:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895127936:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233279.922068:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233276/real 1761233279] req@00000000f4e415d3 x1846780895128000/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233301 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233279.922081:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895128000/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233301 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233279.922089:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233279.922091:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233279.922094:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895128000:192.168.201.157@tcp:38:kworker/u8:2.0 00010000:00010000:1.0:1761233280.689386:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233281.713507:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233281.969441:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233281.969453:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233281.969458:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233281.969471:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5842 00000100:00080000:2.0:1761233281.969493:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233281.969501:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233281.969542:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895128384/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233281.969574:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233281.969579:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233281.969583:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233281.969587:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5842 00000100:00080000:2.0:1761233281.969590:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233281.969594:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233281.969604:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233281.969618:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895128448/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233281.969630:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233281.969634:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233281.969637:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233281.969641:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5842 00000100:00080000:2.0:1761233281.969644:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233281.969650:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895128384:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761233281.969651:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233281.969679:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233281.969697:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895128512/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233281.969710:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233281.969715:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233281.969718:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233281.969723:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5842 00000100:00080000:2.0:1761233281.969725:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233281.969730:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233281.969737:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233281.969752:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895128576/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233281.969764:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1761233281.969772:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233281.969774:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233281.969780:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233281.969787:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5842 00000100:00080000:2.0:1761233281.969791:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233281.969798:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233281.969805:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233281.969810:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233281/real 1761233281] req@00000000fd4aa3a2 x1846780895128384/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233306 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233281.969827:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895128640/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233281.969842:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895128384/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233306 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233281.969848:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233281.969859:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233281.969865:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233281.969869:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233281.969875:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5842 00000100:00080000:3.0:1761233281.969878:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761233281.969883:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233281.969890:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895128384:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761233281.969894:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233281.969908:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233281.969911:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895128448:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233281.969931:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895128704/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233281.969947:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233281.969952:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1761233281.969956:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233281.969958:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233281.969970:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5842 00000100:00080000:2.0:1761233281.969975:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233281.969976:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233281/real 1761233281] req@000000006c1041ba x1846780895128448/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233306 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233281.969987:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233281.970018:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233281.970028:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895128448/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233306 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233281.970045:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895128768/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233281.970084:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233281.970093:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233281.970102:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895128448:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233281.970149:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895128512:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233281.970190:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233281.970207:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233281/real 1761233281] req@000000006d8c3eb0 x1846780895128512/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233306 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233281.970227:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895128512/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233306 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233281.970280:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233281.970285:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233281.970291:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895128512:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233281.970302:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895128576:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233281.970341:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233281.970356:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233281/real 1761233281] req@0000000088939481 x1846780895128576/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233306 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233281.970406:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895128576/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233306 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233281.970419:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233281.970422:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233281.970427:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895128576:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233281.970438:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895128640:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233281.970475:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233281.970492:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233281/real 1761233281] req@000000006ff1fdbc x1846780895128640/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233306 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233281.970511:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895128640/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233306 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233281.970524:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233281.970528:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233281.970533:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895128640:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233281.970543:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895128704:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233281.970577:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233281.970592:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233281/real 1761233281] req@00000000135e9a6e x1846780895128704/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233306 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233281.970609:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895128704/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233306 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233281.970621:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233281.970624:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233281.970629:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895128704:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233281.970638:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895128768:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233281.970671:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233281.970684:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233281/real 1761233281] req@00000000fd3935a6 x1846780895128768/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233306 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233281.970703:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895128768/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233306 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233281.970716:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233281.970719:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233281.970725:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895128768:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233282.737437:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233283.761419:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233284.785423:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233285.809476:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233286.833433:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233287.089387:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233287.089396:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233287.089400:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233287.089409:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5848 00000100:00080000:0.0:1761233287.089412:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233287.089418:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233287.089450:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895128832/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233287.089470:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233287.089473:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233287.089474:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233287.089477:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5848 00000100:00080000:0.0:1761233287.089479:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233287.089482:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233287.089489:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233287.089498:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895128896/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233287.089502:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895128832:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233287.089507:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233287.089510:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233287.089511:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233287.089514:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5848 00000100:00080000:0.0:1761233287.089515:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233287.089517:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233287.089520:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233287.089528:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895128960/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233287.089533:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233287.089536:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233287.089537:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233287.089539:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5848 00000100:00080000:0.0:1761233287.089540:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233287.089542:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233287.089544:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233287.089561:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895129024/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233287.089569:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233287.089572:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233287.089573:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233287.089575:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5848 00000100:00080000:0.0:1761233287.089576:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233287.089578:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233287.089580:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233287.089588:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895129088/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233287.089592:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233287.089597:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233287.089600:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233287.089601:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233287.089605:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5848 00000100:00080000:0.0:1761233287.089606:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233287.089609:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233287.089612:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233287.089618:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233287/real 1761233287] req@00000000c6e06344 x1846780895128832/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233312 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233287.089621:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895129152/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233287.089657:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233287.089660:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233287.089661:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233287.089665:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5848 00000100:00080000:0.0:1761233287.089668:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233287.089668:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895128832/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233312 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233287.089674:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233287.089680:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761233287.089684:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233287.089687:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233287.089688:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895129216/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233287.089693:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895128832:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233287.089718:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895128896:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233287.089733:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233287.089738:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233287/real 1761233287] req@000000005593c994 x1846780895128896/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233312 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233287.089745:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895128896/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233312 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233287.089754:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233287.089756:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233287.089758:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895128896:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233287.089761:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895128960:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233287.089771:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233287.089775:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233287/real 1761233287] req@000000001f9ec359 x1846780895128960/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233312 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233287.089782:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895128960/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233312 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233287.089786:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233287.089787:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233287.089788:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895128960:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233287.089791:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895129024:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233287.089801:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233287.089804:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233287/real 1761233287] req@00000000efa29eb9 x1846780895129024/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233312 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233287.089810:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895129024/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233312 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233287.089815:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233287.089816:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233287.089817:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895129024:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233287.089821:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895129088:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233287.089831:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233287.089835:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233287/real 1761233287] req@000000006d61dd06 x1846780895129088/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233312 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233287.089840:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895129088/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233312 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233287.089845:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233287.089846:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233287.089848:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895129088:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233287.089851:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895129152:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233287.089860:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233287.089863:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233287/real 1761233287] req@00000000ef655201 x1846780895129152/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233312 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233287.089868:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895129152/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233312 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233287.089873:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233287.089874:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233287.089876:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895129152:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233287.089879:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895129216:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233287.089888:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233287.089891:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233287/real 1761233287] req@000000004130b6bd x1846780895129216/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233312 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233287.089898:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895129216/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233312 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233287.089902:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233287.089903:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233287.089904:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895129216:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233287.857581:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233288.881390:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233289.905432:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233290.929402:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233291.953470:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233292.209418:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233292.209431:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233292.209436:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233292.209450:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5853 00000100:00080000:2.0:1761233292.209454:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233292.209462:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233292.209508:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895129280/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233292.209542:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233292.209547:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233292.209549:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233292.209553:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5853 00000100:00080000:2.0:1761233292.209556:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233292.209560:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233292.209571:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233292.209588:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895129344/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233292.209602:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233292.209608:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895129280:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761233292.209609:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233292.209613:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233292.209626:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5853 00000100:00080000:2.0:1761233292.209630:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233292.209633:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233292.209639:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233292.209658:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895129408/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233292.209669:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233292.209673:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233292.209675:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233292.209681:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5853 00000100:00080000:2.0:1761233292.209684:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233292.209688:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233292.209692:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233292.209705:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895129472/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233292.209714:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1761233292.209723:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233292.209724:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233292.209730:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233292.209737:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5853 00000100:00080000:2.0:1761233292.209741:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233292.209746:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233292.209751:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233292.209765:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233292/real 1761233292] req@000000004f29ca26 x1846780895129280/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233317 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233292.209769:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895129536/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233292.209827:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233292.209835:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1761233292.209835:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895129280/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233317 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233292.209841:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233292.209858:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5853 00000100:00080000:3.0:1761233292.209860:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233292.209864:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233292.209867:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761233292.209879:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233292.209881:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895129280:192.168.201.157@tcp:250:kworker/u8:3.0 00000008:00100000:2.0:1761233292.209899:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233292.209923:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895129344:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233292.209928:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895129600/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233292.209961:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233292.209965:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233292.209974:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1761233292.209976:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233292/real 1761233292] req@0000000033765045 x1846780895129344/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233317 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233292.209980:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233292.209997:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5853 00000100:00080000:2.0:1761233292.210003:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233292.210005:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895129344/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233317 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233292.210018:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:3.0:1761233292.210024:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1761233292.210031:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761233292.210032:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233292.210041:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895129344:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233292.210052:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895129408:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233292.210064:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895129664/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233292.210081:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233292.210100:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233292/real 1761233292] req@000000009d315fa5 x1846780895129408/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233317 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233292.210114:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895129408/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233317 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233292.210124:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233292.210126:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233292.210130:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895129408:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233292.210138:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895129472:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233292.210156:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233292.210163:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233292/real 1761233292] req@0000000040d5f361 x1846780895129472/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233317 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233292.210175:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895129472/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233317 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233292.210183:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233292.210185:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233292.210188:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895129472:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233292.210195:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895129536:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233292.210273:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233292.210284:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233292/real 1761233292] req@000000007dcf84ae x1846780895129536/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233317 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233292.210297:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895129536/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233317 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233292.210305:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233292.210308:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233292.210312:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895129536:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233292.210338:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895129600:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233292.210358:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233292.210366:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233292/real 1761233292] req@000000007091f547 x1846780895129600/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233317 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233292.210378:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895129600/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233317 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233292.210386:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233292.210388:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233292.210391:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895129600:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233292.210397:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895129664:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233292.210414:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233292.210421:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233292/real 1761233292] req@0000000064a2cea0 x1846780895129664/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233317 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233292.210434:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895129664/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233317 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233292.210442:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233292.210444:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233292.210447:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895129664:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233292.977480:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233294.001493:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233295.025511:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233296.049451:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233297.073382:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233297.329390:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233297.329407:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233297.329414:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233297.329433:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5858 00000100:00080000:0.0:1761233297.329439:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233297.329449:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233297.329508:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895129728/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233297.329560:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233297.329565:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233297.329567:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233297.329572:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5858 00000100:00080000:0.0:1761233297.329575:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233297.329579:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233297.329594:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233297.329624:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895129792/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233297.329637:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233297.329646:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895129728:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233297.329647:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233297.329652:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233297.329667:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5858 00000100:00080000:0.0:1761233297.329672:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233297.329676:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233297.329682:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233297.329706:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895129856/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233297.329717:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233297.329724:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233297.329728:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233297.329734:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5858 00000100:00080000:0.0:1761233297.329739:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233297.329746:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:3.0:1761233297.329754:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:0.0:1761233297.329756:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233297.329789:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895129920/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233297.329790:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233297/real 1761233297] req@000000007bd1ec64 x1846780895129728/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233322 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233297.329839:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895129728/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233322 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233297.329846:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233297.329859:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233297.329862:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233297.329865:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233297.329869:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233297.329872:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895129728:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233297.329894:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5858 00000100:00080000:0.0:1761233297.329901:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233297.329910:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233297.329919:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895129792:192.168.201.157@tcp:38:kworker/u8:3.0 00000008:00100000:0.0:1761233297.329923:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233297.329953:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233297.329955:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895129984/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233297.329967:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233297/real 1761233297] req@000000001db96463 x1846780895129792/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233322 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233297.330013:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895129792/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233322 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233297.330104:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233297.330108:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233297.330113:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895129792:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:0.0:1761233297.330126:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233297.330126:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895129856:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1761233297.330176:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233297.330181:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1761233297.330193:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233297.330194:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5858 00000100:00100000:3.0:1761233297.330280:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233297/real 1761233297] req@000000009d1c7f73 x1846780895129856/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233322 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233297.330297:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233297.330300:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895129856/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233322 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233297.330314:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:3.0:1761233297.330331:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1761233297.330335:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761233297.330338:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233297.330352:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895129856:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233297.330365:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895129920:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233297.330377:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895130048/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233297.330395:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233297.330408:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233297.330412:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233297/real 1761233297] req@00000000d67bb589 x1846780895129920/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233322 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233297.330426:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1761233297.330435:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895129920/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233322 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233297.330437:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233297.330458:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233297.330460:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5858 00000100:00080000:3.0:1761233297.330463:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233297.330470:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233297.330473:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895129920:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1761233297.330485:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233297.330499:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233297.330503:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895129984:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233297.330542:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895130112/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233297.330544:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233297.330564:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233297/real 1761233297] req@00000000a4a2c44a x1846780895129984/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233322 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233297.330604:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895129984/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233322 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233297.330614:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233297.330617:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233297.330624:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895129984:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233297.330637:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895130048:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233297.330662:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233297.330669:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233297/real 1761233297] req@000000004a74da19 x1846780895130048/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233322 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233297.330678:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895130048/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233322 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233297.330684:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233297.330686:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233297.330689:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895130048:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233297.330704:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895130112:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233297.330718:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233297.330724:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233297/real 1761233297] req@000000002d1b8ef7 x1846780895130112/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233322 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233297.330734:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895130112/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233322 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233297.330740:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233297.330742:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233297.330744:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895130112:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233298.097410:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233299.121343:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233300.145452:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233301.169396:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233302.193370:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233302.449350:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233302.449363:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233302.449368:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233302.449380:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5863 00000100:00080000:0.0:1761233302.449385:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233302.449394:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233302.449436:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895130176/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233302.449466:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233302.449471:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233302.449473:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233302.449477:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5863 00000100:00080000:0.0:1761233302.449480:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233302.449486:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233302.449498:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233302.449501:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895130176:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233302.449515:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895130240/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233302.449528:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233302.449532:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233302.449534:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233302.449539:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5863 00000100:00080000:0.0:1761233302.449541:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233302.449545:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233302.449551:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233302.449565:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895130304/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233302.449572:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233302.449590:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233302.449595:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233302.449598:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233302.449598:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233302/real 1761233302] req@00000000f3ed2588 x1846780895130176/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233327 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233302.449608:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5863 00000100:00080000:0.0:1761233302.449614:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233302.449620:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233302.449625:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233302.449628:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f3ed2588 x1846780895130176/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233327 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233302.449639:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895130368/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233302.449639:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233302.449645:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233302.449653:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895130176:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233302.449654:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233302.449661:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233302.449662:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233302.449667:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5863 00000100:00080000:0.0:1761233302.449668:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233302.449672:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233302.449675:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233302.449684:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895130240:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761233302.449686:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895130432/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233302.449700:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233302.449703:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233302.449705:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233302.449709:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5863 00000100:00080000:0.0:1761233302.449712:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1761233302.449715:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233302.449721:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233302.449726:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233302/real 1761233302] req@000000005d83493f x1846780895130240/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233327 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000008:00100000:0.0:1761233302.449729:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233302.449742:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005d83493f x1846780895130240/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233327 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233302.449749:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1761233302.449752:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895130496/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233302.449753:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233302.449761:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895130240:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233302.449767:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895130304:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233302.449803:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233302.449810:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233302/real 1761233302] req@000000002d1b8ef7 x1846780895130304/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233327 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233302.449811:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233302.449820:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233302.449823:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233302.449825:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895130304/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233327 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233302.449831:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5863 00000100:00080000:3.0:1761233302.449833:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233302.449836:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233302.449836:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233302.449842:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895130304:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00080000:0.0:1761233302.449845:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233302.449849:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895130368:192.168.201.157@tcp:8:kworker/u8:2.0 00000008:00100000:0.0:1761233302.449853:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233302.449863:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233302.449870:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895130560/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233302.449886:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233302/real 1761233302] req@000000004a74da19 x1846780895130368/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233327 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233302.449901:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895130368/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233327 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233302.449909:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233302.449911:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233302.449914:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895130368:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233302.449936:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895130432:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233302.449952:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233302.449958:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233302/real 1761233302] req@00000000a4a2c44a x1846780895130432/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233327 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233302.449965:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895130432/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233327 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233302.449969:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233302.449971:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233302.449973:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895130432:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233302.449976:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895130496:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233302.449985:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233302.449989:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233302/real 1761233302] req@00000000d67bb589 x1846780895130496/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233327 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233302.449995:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895130496/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233327 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233302.449999:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233302.450000:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233302.450002:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895130496:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233302.450005:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895130560:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233302.450013:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233302.450016:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233302/real 1761233302] req@000000009d1c7f73 x1846780895130560/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233327 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233302.450023:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895130560/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233327 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233302.450027:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233302.450028:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233302.450029:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895130560:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233303.217452:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233304.241371:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233305.265420:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233306.289484:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233307.313471:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233307.569375:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233307.569391:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233307.569396:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233307.569422:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5868 00000100:00080000:0.0:1761233307.569427:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233307.569437:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233307.569496:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895130624/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233307.569534:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233307.569539:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233307.569541:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233307.569545:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5868 00000100:00080000:0.0:1761233307.569547:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233307.569551:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233307.569566:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233307.569578:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895130688/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233307.569589:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233307.569594:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233307.569598:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233307.569602:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5868 00000100:00080000:0.0:1761233307.569604:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233307.569608:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233307.569612:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233307.569621:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895130752/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233307.569632:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895130624:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233307.569638:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233307.569648:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233307.569650:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233307.569655:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5868 00000100:00080000:0.0:1761233307.569657:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233307.569663:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233307.569685:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233307.569703:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895130816/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233307.569714:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233307.569718:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233307.569721:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233307.569726:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5868 00000100:00080000:0.0:1761233307.569730:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1761233307.569738:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233307.569742:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233307.569751:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233307.569786:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895130880/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233307.569787:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233307/real 1761233307] req@000000001db96463 x1846780895130624/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233332 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233307.569881:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233307.569887:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233307.569890:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233307.569890:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895130624/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233332 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233307.569912:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5868 00000100:00080000:0.0:1761233307.569922:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233307.569936:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233307.569937:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:3.0:1761233307.569949:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000008:00100000:0.0:1761233307.569951:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233307.570017:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895130624:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233307.570027:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895130944/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233307.570032:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895130688:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:0.0:1761233307.570063:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233307.570067:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233307.570071:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233307.570078:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5868 00000100:00080000:0.0:1761233307.570093:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233307.570098:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:3.0:1761233307.570109:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:0.0:1761233307.570113:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233307.570124:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233307/real 1761233307] req@000000007bd1ec64 x1846780895130688/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233332 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233307.570133:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895131008/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233307.570161:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895130688/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233332 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233307.570203:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233307.570205:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233307.570211:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895130688:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233307.570295:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895130752:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233307.570316:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233307.570322:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233307/real 1761233307] req@000000004130b6bd x1846780895130752/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233332 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233307.570330:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895130752/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233332 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233307.570335:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233307.570337:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233307.570339:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895130752:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233307.570344:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895130816:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233307.570354:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233307.570359:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233307/real 1761233307] req@00000000ef655201 x1846780895130816/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233332 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233307.570366:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895130816/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233332 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233307.570371:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233307.570372:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233307.570374:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895130816:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233307.570379:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895130880:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233307.570389:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233307.570394:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233307/real 1761233307] req@000000006d61dd06 x1846780895130880/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233332 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233307.570400:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895130880/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233332 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233307.570405:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233307.570407:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233307.570409:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895130880:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233307.570412:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895130944:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233307.570422:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233307.570427:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233307/real 1761233307] req@00000000efa29eb9 x1846780895130944/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233332 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233307.570433:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895130944/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233332 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233307.570437:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233307.570439:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233307.570440:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895130944:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233307.570444:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895131008:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233307.570454:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233307.570458:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233307/real 1761233307] req@000000001f9ec359 x1846780895131008/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233332 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233307.570466:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895131008/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233332 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233307.570471:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233307.570473:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233307.570474:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895131008:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233308.337418:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233309.361432:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233310.385485:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233311.409424:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233312.433339:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233312.689400:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233312.689415:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233312.689422:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233312.689437:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5873 00000100:00080000:1.0:1761233312.689443:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233312.689453:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233312.689499:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000880f16b4 x1846780895131072/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233312.689533:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233312.689537:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233312.689539:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233312.689544:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5873 00000100:00080000:1.0:1761233312.689546:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233312.689550:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233312.689561:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233312.689576:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000053e52331 x1846780895131136/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233312.689588:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233312.689594:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233312.689597:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233312.689600:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895131072:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233312.689607:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5873 00000100:00080000:1.0:1761233312.689617:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233312.689629:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233312.689636:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233312.689653:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090ac513e x1846780895131200/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233312.689664:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233312.689669:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233312.689671:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233312.689675:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5873 00000100:00080000:1.0:1761233312.689678:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233312.689681:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233312.689687:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233312.689701:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d66930c2 x1846780895131264/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233312.689708:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:1.0:1761233312.689720:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233312.689725:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233312.689728:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233312.689735:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5873 00000100:00080000:1.0:1761233312.689737:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233312.689743:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233312.689749:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233312/real 1761233312] req@00000000880f16b4 x1846780895131072/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233337 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:1.0:1761233312.689753:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233312.689772:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ffe86d04 x1846780895131328/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233312.689783:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000880f16b4 x1846780895131072/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233337 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233312.689791:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233312.689802:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233312.689805:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233312.689813:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233312.689816:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5873 00000100:00080000:3.0:1761233312.689820:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:1.0:1761233312.689822:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233312.689833:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233312.689835:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895131072:192.168.201.157@tcp:250:kworker/u8:3.0 00000008:00100000:1.0:1761233312.689839:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233312.689857:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df5b2270 x1846780895131392/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233312.689867:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233312.689871:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233312.689873:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233312.689878:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895131136:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:1.0:1761233312.689885:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5873 00000100:00080000:1.0:1761233312.689891:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233312.689930:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233312.689939:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233312.689946:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233312.689963:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233312/real 1761233312] req@0000000053e52331 x1846780895131136/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233337 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233312.689969:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df0b94a8 x1846780895131456/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233312.690015:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000053e52331 x1846780895131136/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233337 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233312.690029:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233312.690034:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233312.690039:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895131136:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233312.690047:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895131200:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233312.690072:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233312.690080:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233312/real 1761233312] req@0000000090ac513e x1846780895131200/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233337 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233312.690093:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090ac513e x1846780895131200/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233337 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233312.690101:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233312.690103:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233312.690106:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895131200:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233312.690112:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895131264:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233312.690130:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233312.690137:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233312/real 1761233312] req@00000000d66930c2 x1846780895131264/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233337 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233312.690149:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d66930c2 x1846780895131264/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233337 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233312.690157:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233312.690159:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233312.690162:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895131264:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233312.690169:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895131328:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233312.690187:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233312.690195:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233312/real 1761233312] req@00000000ffe86d04 x1846780895131328/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233337 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233312.690241:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ffe86d04 x1846780895131328/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233337 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233312.690250:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233312.690253:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233312.690256:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895131328:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233312.690263:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895131392:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233312.690285:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233312.690294:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233312/real 1761233312] req@00000000df5b2270 x1846780895131392/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233337 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233312.690306:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df5b2270 x1846780895131392/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233337 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233312.690314:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233312.690316:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233312.690319:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895131392:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233312.690346:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895131456:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233312.690365:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233312.690373:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233312/real 1761233312] req@00000000df0b94a8 x1846780895131456/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233337 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233312.690386:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df0b94a8 x1846780895131456/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233337 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233312.690394:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233312.690396:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233312.690399:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895131456:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233313.457458:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233314.481361:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233315.505444:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233316.529435:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233317.553502:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233317.809421:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233317.809431:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233317.809434:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233317.809445:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5878 00000100:00080000:0.0:1761233317.809448:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233317.809454:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233317.809491:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895131520/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233317.809522:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233317.809527:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233317.809529:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233317.809550:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5878 00000100:00080000:0.0:1761233317.809554:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233317.809563:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233317.809579:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233317.809582:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895131520:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761233317.809608:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895131584/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233317.809630:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233317.809636:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233317.809639:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233317.809645:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5878 00000100:00080000:0.0:1761233317.809649:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233317.809656:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233317.809669:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233317.809675:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233317.809697:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895131648/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233317.809709:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233317/real 1761233317] req@000000005593c994 x1846780895131520/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233342 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233317.809725:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233317.809734:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233317.809737:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233317.809739:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895131520/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233342 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233317.809754:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5878 00000100:00080000:3.0:1761233317.809761:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233317.809762:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233317.809770:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233317.809776:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233317.809782:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895131520:192.168.201.157@tcp:250:kworker/u8:3.0 00000008:00100000:0.0:1761233317.809791:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233317.809838:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895131584:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233317.809845:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895131712/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233317.809878:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233317.809882:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:3.0:1761233317.809886:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233317.809887:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233317.809894:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5878 00000100:00080000:0.0:1761233317.809898:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233317.809898:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233317/real 1761233317] req@00000000c6e06344 x1846780895131584/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233342 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233317.809908:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233317.809913:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895131584/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233342 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:0.0:1761233317.809915:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761233317.809930:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233317.809933:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233317.809935:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895131776/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233317.809938:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895131584:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233317.809943:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895131648:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233317.809959:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233317.809961:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233317.809967:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1761233317.809967:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233317/real 1761233317] req@000000004862b808 x1846780895131648/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233342 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233317.809971:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233317.809981:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5878 00000100:00080000:0.0:1761233317.809984:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233317.809984:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895131648/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233342 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233317.809991:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233317.809995:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233317.809999:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233317.810000:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895131648:192.168.201.157@tcp:8:kworker/u8:3.0 00000008:00100000:0.0:1761233317.810010:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233317.810017:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895131712:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233317.810028:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895131840/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233317.810035:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233317.810043:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233317/real 1761233317] req@000000006f2efc8b x1846780895131712/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233342 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233317.810046:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233317.810057:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233317.810077:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233317.810080:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895131712/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233342 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233317.810090:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5878 00000100:00080000:3.0:1761233317.810097:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233317.810099:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233317.810103:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233317.810113:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233317.810114:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895131712:192.168.201.157@tcp:8:kworker/u8:3.0 00000008:00100000:0.0:1761233317.810128:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233317.810130:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895131776:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233317.810161:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895131904/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233317.810173:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233317.810186:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233317/real 1761233317] req@00000000eca9ddc2 x1846780895131776/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233342 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233317.810200:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895131776/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233342 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233317.810209:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233317.810212:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233317.810215:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895131776:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233317.810276:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895131840:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233317.810292:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233317.810297:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233317/real 1761233317] req@0000000034d49f59 x1846780895131840/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233342 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233317.810305:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895131840/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233342 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233317.810310:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233317.810311:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233317.810313:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895131840:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233317.810317:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895131904:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233317.810327:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233317.810331:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233317/real 1761233317] req@000000005beaae31 x1846780895131904/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233342 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233317.810339:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895131904/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233342 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233317.810344:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233317.810345:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233317.810347:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895131904:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233318.577475:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233319.601534:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233320.625509:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233321.649449:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233322.673471:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233322.929294:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233322.929305:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233322.929309:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233322.929319:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5883 00000100:00080000:0.0:1761233322.929322:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233322.929328:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233322.929362:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000addc3db8 x1846780895131968/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233322.929385:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233322.929389:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233322.929390:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233322.929393:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5883 00000100:00080000:0.0:1761233322.929394:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233322.929397:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233322.929404:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233322.929413:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000078e7052d x1846780895132032/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233322.929429:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233322.929431:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895131968:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233322.929433:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233322.929435:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233322.929441:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5883 00000100:00080000:0.0:1761233322.929443:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233322.929446:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233322.929451:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233322.929464:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895132096/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233322.929473:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233322.929477:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233322.929479:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233322.929483:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5883 00000100:00080000:0.0:1761233322.929485:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233322.929489:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233322.929494:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233322.929503:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233322.929508:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895132160/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233322.929515:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233322.929518:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233322.929520:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233322.929523:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5883 00000100:00080000:0.0:1761233322.929525:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233322.929528:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233322/real 1761233322] req@00000000addc3db8 x1846780895131968/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233347 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233322.929531:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233322.929538:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233322.929550:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895132224/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233322.929554:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000addc3db8 x1846780895131968/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233347 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233322.929570:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233322.929575:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233322.929575:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233322.929578:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233322.929579:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233322.929586:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895131968:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233322.929589:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5883 00000100:00080000:0.0:1761233322.929593:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233322.929597:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233322.929603:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233322.929610:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895132032:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233322.929617:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895132288/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233322.929628:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1761233322.929634:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233322.929635:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233322.929638:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233322.929644:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5883 00000100:00100000:3.0:1761233322.929646:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233322/real 1761233322] req@0000000078e7052d x1846780895132032/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233347 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233322.929647:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233322.929654:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233322.929661:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233322.929665:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000078e7052d x1846780895132032/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233347 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233322.929673:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895132352/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233322.929674:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233322.929679:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233322.929686:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895132032:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233322.929691:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895132096:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233322.929718:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233322.929730:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233322/real 1761233322] req@000000005beaae31 x1846780895132096/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233347 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233322.929793:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895132096/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233347 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233322.929801:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233322.929804:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233322.929807:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895132096:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233322.929813:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895132160:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233322.929834:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233322.929843:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233322/real 1761233322] req@0000000034d49f59 x1846780895132160/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233347 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233322.929853:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895132160/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233347 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233322.929860:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233322.929862:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233322.929865:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895132160:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233322.929871:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895132224:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233322.929890:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233322.929897:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233322/real 1761233322] req@00000000eca9ddc2 x1846780895132224/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233347 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233322.929908:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895132224/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233347 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233322.929916:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233322.929918:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233322.929921:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895132224:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233322.929940:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895132288:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233322.929962:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233322.929971:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233322/real 1761233322] req@000000006f2efc8b x1846780895132288/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233347 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233322.929982:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895132288/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233347 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233322.929989:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233322.930002:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233322.930006:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895132288:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233322.930011:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895132352:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233322.930033:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233322.930043:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233322/real 1761233322] req@000000004862b808 x1846780895132352/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233347 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233322.930056:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895132352/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233347 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233322.930063:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233322.930065:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233322.930067:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895132352:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233323.697444:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233324.721470:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233325.745420:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233326.769468:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233327.793383:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233328.049355:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233328.049365:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233328.049368:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233328.049377:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5888 00000100:00080000:2.0:1761233328.049381:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233328.049387:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233328.049420:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895132416/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233328.049438:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233328.049440:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233328.049442:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233328.049444:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5888 00000100:00080000:2.0:1761233328.049445:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233328.049447:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233328.049457:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233328.049466:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895132480/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233328.049473:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233328.049476:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895132416:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761233328.049477:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233328.049479:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233328.049484:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5888 00000100:00080000:2.0:1761233328.049486:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233328.049489:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233328.049492:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233328.049501:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895132544/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233328.049506:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233328.049509:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233328.049510:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233328.049512:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5888 00000100:00080000:2.0:1761233328.049513:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233328.049515:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233328.049518:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233328.049524:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895132608/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233328.049528:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233328.049530:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233328.049531:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233328.049534:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5888 00000100:00080000:2.0:1761233328.049535:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233328.049538:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233328.049540:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233328.049544:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233328.049548:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895132672/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233328.049554:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233328.049556:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233328.049558:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233328.049561:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5888 00000100:00080000:2.0:1761233328.049562:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233328.049565:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233328.049568:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233328/real 1761233328] req@0000000064a2cea0 x1846780895132416/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233353 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000008:00100000:2.0:1761233328.049569:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233328.049578:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895132736/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233328.049584:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233328.049587:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1761233328.049587:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895132416/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233353 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233328.049588:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233328.049594:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5888 00000100:00080000:2.0:1761233328.049598:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233328.049602:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:3.0:1761233328.049604:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1761233328.049605:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761233328.049609:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233328.049615:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895132416:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761233328.049616:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895132800/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233328.049641:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895132480:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233328.049681:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233328.049687:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233328/real 1761233328] req@000000007091f547 x1846780895132480/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233353 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233328.049695:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895132480/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233353 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233328.049704:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233328.049706:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233328.049708:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895132480:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233328.049711:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895132544:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233328.049722:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233328.049726:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233328/real 1761233328] req@000000007dcf84ae x1846780895132544/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233353 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233328.049732:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895132544/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233353 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233328.049736:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233328.049737:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233328.049738:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895132544:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233328.049741:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895132608:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233328.049751:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233328.049755:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233328/real 1761233328] req@0000000040d5f361 x1846780895132608/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233353 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233328.049761:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895132608/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233353 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233328.049765:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233328.049767:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233328.049768:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895132608:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233328.049772:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895132672:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233328.049781:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233328.049785:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233328/real 1761233328] req@000000009d315fa5 x1846780895132672/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233353 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233328.049791:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895132672/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233353 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233328.049796:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233328.049797:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233328.049798:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895132672:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233328.049801:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895132736:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233328.049810:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233328.049814:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233328/real 1761233328] req@0000000033765045 x1846780895132736/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233353 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233328.049820:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895132736/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233353 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233328.049823:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233328.049825:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233328.049826:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895132736:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233328.049829:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895132800:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233328.049838:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233328.049842:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233328/real 1761233328] req@000000004f29ca26 x1846780895132800/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233353 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233328.049848:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895132800/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233353 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233328.049853:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233328.049854:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233328.049855:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895132800:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233328.817410:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233329.841413:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233330.865436:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233331.889379:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233332.913567:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233333.169347:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233333.169360:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233333.169365:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233333.169379:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5894 00000100:00080000:2.0:1761233333.169384:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233333.169391:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233333.169441:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895132864/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233333.169480:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233333.169484:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233333.169486:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233333.169490:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5894 00000100:00080000:2.0:1761233333.169493:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233333.169497:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233333.169507:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233333.169521:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895132928/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233333.169531:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233333.169536:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233333.169537:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233333.169542:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5894 00000100:00080000:2.0:1761233333.169544:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233333.169547:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233333.169557:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233333.169567:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895132992/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233333.169604:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233333.169608:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233333.169610:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233333.169615:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5894 00000100:00080000:2.0:1761233333.169617:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233333.169621:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233333.169626:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233333.169645:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895133056/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233333.169647:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895132864:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761233333.169682:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233333.169686:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233333.169688:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233333.169693:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5894 00000100:00080000:2.0:1761233333.169697:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233333.169703:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233333.169711:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233333.169724:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895133120/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233333.169733:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233333.169738:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233333.169739:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233333.169744:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5894 00000100:00080000:2.0:1761233333.169747:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233333.169753:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233333.169758:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233333.169767:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895133184/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233333.169775:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233333.169778:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233333.169781:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233333.169785:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5894 00000100:00080000:2.0:1761233333.169787:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1761233333.169794:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233333.169802:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233333.169809:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233333.169824:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895133248/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233333.169847:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233333/real 1761233333] req@00000000fd3935a6 x1846780895132864/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233358 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233333.169876:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895132864/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233358 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233333.169890:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233333.169894:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233333.169899:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895132864:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233333.169908:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895132928:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761233333.169945:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233333.169959:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233333/real 1761233333] req@00000000135e9a6e x1846780895132928/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233358 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233333.169979:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895132928/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233358 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233333.169992:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233333.169996:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233333.170001:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895132928:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761233333.170013:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895132992:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233333.170049:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233333.170062:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233333/real 1761233333] req@000000006ff1fdbc x1846780895132992/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233358 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233333.170075:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895132992/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233358 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233333.170084:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233333.170086:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233333.170089:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895132992:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233333.170157:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895133056:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233333.170190:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233333.170205:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233333/real 1761233333] req@0000000088939481 x1846780895133056/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233358 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233333.170320:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895133056/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233358 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233333.170336:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233333.170340:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233333.170346:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895133056:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233333.170358:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895133120:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761233333.170398:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233333.170410:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233333/real 1761233333] req@000000006d8c3eb0 x1846780895133120/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233358 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233333.170426:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895133120/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233358 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233333.170435:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233333.170438:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233333.170442:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895133120:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761233333.170448:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895133184:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233333.170469:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233333.170477:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233333/real 1761233333] req@000000006c1041ba x1846780895133184/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233358 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233333.170496:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895133184/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233358 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233333.170505:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233333.170507:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233333.170510:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895133184:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233333.170516:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895133248:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233333.170534:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233333.170541:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233333/real 1761233333] req@00000000fd4aa3a2 x1846780895133248/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233358 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233333.170552:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895133248/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233358 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233333.170561:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233333.170564:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233333.170567:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895133248:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233333.937361:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233334.961388:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233335.985366:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233337.009421:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233338.033426:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233338.289357:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233338.289366:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233338.289369:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233338.289379:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5899 00000100:00080000:0.0:1761233338.289382:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233338.289388:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233338.289421:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895133312/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233338.289446:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233338.289449:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233338.289450:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233338.289452:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5899 00000100:00080000:0.0:1761233338.289454:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233338.289456:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233338.289463:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233338.289471:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895133376/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233338.289480:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233338.289483:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233338.289484:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233338.289486:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5899 00000100:00080000:0.0:1761233338.289488:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233338.289511:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233338.289517:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233338.289526:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895133312:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761233338.289531:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895133440/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233338.289546:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233338.289550:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233338.289552:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233338.289557:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5899 00000100:00080000:0.0:1761233338.289560:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233338.289564:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233338.289569:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233338.289583:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895133504/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233338.289592:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233338.289595:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233338.289595:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233338.289598:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5899 00000100:00080000:0.0:1761233338.289599:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233338.289602:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233338.289605:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233338.289612:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895133568/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233338.289617:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233338.289620:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233338.289622:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1761233338.289629:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233338.289632:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5899 00000100:00080000:0.0:1761233338.289636:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233338.289639:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233338.289643:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233338.289652:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895133632/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233338.289657:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233338.289660:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233338.289663:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233338.289667:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233338/real 1761233338] req@00000000c6e06344 x1846780895133312/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233363 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233338.289674:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5899 00000100:00080000:0.0:1761233338.289681:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233338.289690:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233338.289702:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233338.289712:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895133312/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233363 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233338.289716:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895133696/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233338.289743:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233338.289749:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233338.289756:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895133312:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233338.289787:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895133376:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233338.289805:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233338.289811:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233338/real 1761233338] req@000000005593c994 x1846780895133376/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233363 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233338.289824:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895133376/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233363 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233338.289830:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233338.289832:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233338.289834:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895133376:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233338.289837:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895133440:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233338.289851:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233338.289855:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233338/real 1761233338] req@000000001f9ec359 x1846780895133440/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233363 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233338.289863:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895133440/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233363 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233338.289868:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233338.289870:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233338.289871:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895133440:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233338.289875:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895133504:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233338.289887:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233338.289891:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233338/real 1761233338] req@00000000efa29eb9 x1846780895133504/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233363 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233338.289898:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895133504/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233363 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233338.289903:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233338.289905:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233338.289907:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895133504:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233338.289912:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895133568:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233338.289923:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233338.289928:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233338/real 1761233338] req@000000006d61dd06 x1846780895133568/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233363 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233338.289935:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895133568/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233363 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233338.289940:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233338.289942:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233338.289944:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895133568:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233338.289948:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895133632:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233338.289959:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233338.289963:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233338/real 1761233338] req@00000000ef655201 x1846780895133632/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233363 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233338.289969:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895133632/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233363 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233338.289974:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233338.289976:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233338.289978:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895133632:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233338.289981:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895133696:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233338.289995:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233338.289999:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233338/real 1761233338] req@000000004130b6bd x1846780895133696/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233363 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233338.290007:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895133696/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233363 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233338.290012:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233338.290014:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233338.290016:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895133696:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233339.057443:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233340.081424:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233341.105415:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233342.129419:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233343.153410:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233343.409346:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233343.409355:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233343.409359:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233343.409381:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5904 00000100:00080000:0.0:1761233343.409384:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233343.409392:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233343.409454:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895133760/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233343.409478:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233343.409481:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233343.409482:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233343.409485:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5904 00000100:00080000:0.0:1761233343.409487:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233343.409490:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233343.409498:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233343.409519:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895133824/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233343.409526:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233343.409529:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233343.409530:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233343.409532:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5904 00000100:00080000:0.0:1761233343.409535:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233343.409536:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233343.409539:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233343.409550:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895133888/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233343.409560:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895133760:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233343.409564:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233343.409582:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233343.409583:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233343.409586:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5904 00000100:00080000:0.0:1761233343.409588:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233343.409590:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233343.409593:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233343.409604:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895133952/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233343.409610:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233343.409613:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233343.409614:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233343.409618:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5904 00000100:00080000:0.0:1761233343.409619:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233343.409622:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233343.409625:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233343.409642:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895134016/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233343.409647:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233343.409649:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233343.409650:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233343.409653:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5904 00000100:00080000:0.0:1761233343.409656:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233343.409660:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233343.409664:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233343.409670:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895134080/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233343.409675:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233343.409677:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233343.409678:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233343.409680:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5904 00000100:00080000:0.0:1761233343.409681:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233343.409684:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233343.409687:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233343.409701:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895134144/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233343.409727:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895133824:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233343.409795:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895133888:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233343.409827:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895133952:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233343.409849:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895134016:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233343.409867:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895134080:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233343.409885:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895134144:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233344.177430:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233345.201358:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233346.225425:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1761233346.481546:0:2208:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:1.0:1761233346.481593:0:2208:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1761233346.481608:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233346.481650:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233346.481660:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233346.481670:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233346.481681:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233346.481692:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:3.0:1761233346.481697:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233343/real 1761233346] req@000000009d1c7f73 x1846780895133888/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233368 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000800:00000100:1.0:1761233346.481705:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:3.0:1761233346.481737:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895133888/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233368 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233346.481748:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233346.481753:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233346.481757:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895133888:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233346.481764:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233343/real 1761233346] req@00000000d67bb589 x1846780895133952/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233368 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233346.481776:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895133952/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233368 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233346.481783:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233346.481785:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233346.481788:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895133952:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233346.481794:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233343/real 1761233346] req@00000000a4a2c44a x1846780895134016/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233368 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233346.481804:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895134016/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233368 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233346.481811:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233346.481814:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233346.481817:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895134016:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233346.481822:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233343/real 1761233346] req@000000004a74da19 x1846780895134080/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233368 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233346.481839:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895134080/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233368 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233346.481846:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233346.481848:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233346.481851:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895134080:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233346.481856:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233343/real 1761233346] req@000000002d1b8ef7 x1846780895134144/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233368 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233346.481865:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895134144/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233368 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233346.481871:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233346.481873:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233346.481876:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895134144:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233346.481881:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233343/real 1761233346] req@000000007bd1ec64 x1846780895133760/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233368 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233346.481891:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895133760/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233368 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233346.481897:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233346.481900:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233346.481902:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895133760:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233346.481907:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233343/real 1761233346] req@000000001db96463 x1846780895133824/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233368 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233346.481923:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895133824/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233368 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233346.481930:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233346.481932:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233346.481936:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895133824:192.168.201.157@tcp:38:kworker/u8:2.0 00010000:00010000:3.0:1761233347.249392:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233348.273498:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233348.529394:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233348.529404:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233348.529408:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233348.529419:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5909 00000100:00080000:2.0:1761233348.529422:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233348.529428:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233348.529461:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895134208/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233348.529488:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233348.529492:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233348.529494:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233348.529497:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5909 00000100:00080000:2.0:1761233348.529499:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233348.529502:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233348.529509:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233348.529520:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895134272/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233348.529529:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233348.529531:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233348.529533:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233348.529538:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5909 00000100:00080000:2.0:1761233348.529540:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233348.529541:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895134208:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761233348.529544:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233348.529551:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233348.529559:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895134336/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233348.529568:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233348.529572:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233348.529574:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233348.529579:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5909 00000100:00080000:2.0:1761233348.529581:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233348.529585:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233348.529591:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233348.529606:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895134400/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233348.529617:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233348.529621:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233348.529623:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233348.529628:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5909 00000100:00080000:2.0:1761233348.529631:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1761233348.529634:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233348.529639:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233348.529647:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233348.529661:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895134464/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233348.529666:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233348/real 1761233348] req@00000000f4e415d3 x1846780895134208/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233373 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233348.529683:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233348.529690:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233348.529692:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233348.529700:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895134208/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233373 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233348.529701:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5909 00000100:00080000:2.0:1761233348.529714:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233348.529718:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233348.529723:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:3.0:1761233348.529725:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000008:00100000:2.0:1761233348.529732:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233348.529736:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895134208:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761233348.529752:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895134528/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233348.529765:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233348.529769:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233348.529772:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233348.529772:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895134272:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:2.0:1761233348.529782:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5909 00000100:00080000:2.0:1761233348.529786:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233348.529792:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233348.529798:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233348.529814:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233348.529815:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895134592/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233348.529832:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233348/real 1761233348] req@0000000099b7e5d9 x1846780895134272/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233373 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233348.529849:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895134272/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233373 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233348.529865:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233348.529868:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233348.529872:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895134272:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233348.529878:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895134336:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233348.529905:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233348.529913:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233348/real 1761233348] req@00000000c10cebe6 x1846780895134336/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233373 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233348.529925:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895134336/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233373 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233348.529933:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233348.529936:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233348.529940:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895134336:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233348.529946:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895134400:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233348.529969:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233348.529979:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233348/real 1761233348] req@00000000a1e8c6bb x1846780895134400/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233373 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233348.529991:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895134400/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233373 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233348.529999:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233348.530001:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233348.530016:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895134400:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233348.530022:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895134464:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233348.530043:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233348.530051:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233348/real 1761233348] req@000000008ebccab5 x1846780895134464/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233373 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233348.530062:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895134464/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233373 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233348.530070:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233348.530073:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233348.530076:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895134464:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233348.530082:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895134528:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233348.530104:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233348.530112:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233348/real 1761233348] req@000000004ae5bc9b x1846780895134528/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233373 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233348.530125:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895134528/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233373 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233348.530133:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233348.530136:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233348.530140:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895134528:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233348.530181:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895134592:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233348.530236:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233348.530248:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233348/real 1761233348] req@00000000ac7d8d5b x1846780895134592/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233373 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233348.530261:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895134592/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233373 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233348.530269:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233348.530272:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233348.530275:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895134592:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233349.297450:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233350.321548:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233351.345431:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233352.369487:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233353.393465:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233353.649352:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233353.649361:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233353.649364:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233353.649373:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5914 00000100:00080000:0.0:1761233353.649376:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233353.649383:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233353.649413:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895134656/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233353.649435:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233353.649438:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233353.649439:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233353.649441:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5914 00000100:00080000:0.0:1761233353.649443:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233353.649445:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233353.649451:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233353.649458:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895134720/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233353.649464:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233353.649467:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233353.649469:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233353.649471:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5914 00000100:00080000:0.0:1761233353.649473:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233353.649474:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233353.649478:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233353.649483:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895134656:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761233353.649487:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895134784/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233353.649496:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233353.649497:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233353.649498:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233353.649500:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5914 00000100:00080000:0.0:1761233353.649501:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233353.649503:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233353.649505:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233353.649513:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895134848/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233353.649517:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233353.649520:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233353.649520:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233353.649523:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5914 00000100:00080000:0.0:1761233353.649524:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233353.649526:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233353.649529:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233353.649534:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895134912/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233353.649540:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233353.649542:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233353.649543:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233353.649545:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5914 00000100:00080000:0.0:1761233353.649546:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233353.649548:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233353.649550:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233353.649556:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895134976/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233353.649561:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233353.649564:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233353.649565:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1761233353.649566:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233353.649570:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5914 00000100:00080000:0.0:1761233353.649571:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233353.649574:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233353.649576:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233353.649583:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895135040/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233353.649596:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233353/real 1761233353] req@000000005d83493f x1846780895134656/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233378 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233353.649616:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005d83493f x1846780895134656/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233378 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233353.649626:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233353.649629:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233353.649633:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895134656:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233353.649640:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895134720:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233353.649664:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233353.649672:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233353/real 1761233353] req@00000000f3ed2588 x1846780895134720/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233378 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233353.649683:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f3ed2588 x1846780895134720/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233378 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233353.649691:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233353.649693:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233353.649696:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895134720:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233353.649721:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895134784:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233353.649741:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233353.649749:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233353/real 1761233353] req@000000001db96463 x1846780895134784/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233378 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233353.649760:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895134784/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233378 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233353.649767:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233353.649769:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233353.649772:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895134784:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233353.649778:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895134848:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233353.649797:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233353.649805:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233353/real 1761233353] req@000000007bd1ec64 x1846780895134848/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233378 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233353.649816:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895134848/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233378 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233353.649823:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233353.649825:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233353.649828:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895134848:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233353.649834:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895134912:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233353.649851:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233353.649858:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233353/real 1761233353] req@000000002d1b8ef7 x1846780895134912/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233378 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233353.649867:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895134912/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233378 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233353.649873:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233353.649875:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233353.649878:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895134912:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233353.649883:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895134976:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233353.649898:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233353.649904:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233353/real 1761233353] req@000000004a74da19 x1846780895134976/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233378 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233353.649914:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895134976/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233378 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233353.649920:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233353.649922:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233353.649925:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895134976:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233353.649930:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895135040:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233353.649945:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233353.649951:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233353/real 1761233353] req@00000000a4a2c44a x1846780895135040/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233378 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233353.649961:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895135040/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233378 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233353.649968:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233353.649970:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233353.649973:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895135040:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233354.417467:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233355.441488:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233356.465443:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233357.489411:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233358.513459:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233358.769405:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233358.769420:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233358.769426:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233358.769441:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5919 00000100:00080000:0.0:1761233358.769446:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233358.769458:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233358.769506:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895135104/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233358.769545:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233358.769559:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233358.769561:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233358.769565:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5919 00000100:00080000:0.0:1761233358.769567:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233358.769572:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233358.769581:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233358.769594:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895135168/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233358.769603:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233358.769607:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233358.769608:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233358.769612:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5919 00000100:00080000:0.0:1761233358.769623:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233358.769627:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233358.769631:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233358.769639:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895135232/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233358.769648:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233358.769652:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233358.769654:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233358.769660:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895135104:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233358.769662:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5919 00000100:00080000:0.0:1761233358.769668:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233358.769674:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233358.769678:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233358.769687:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895135296/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233358.769695:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233358.769698:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233358.769699:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233358.769702:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5919 00000100:00080000:0.0:1761233358.769703:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233358.769707:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233358.769710:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233358.769720:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895135360/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233358.769726:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233358.769729:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233358.769730:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233358.769733:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5919 00000100:00080000:0.0:1761233358.769735:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233358.769739:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233358.769741:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233358.769751:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895135424/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233358.769766:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1761233358.769768:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233358.769770:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233358.769774:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233358.769779:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5919 00000100:00080000:0.0:1761233358.769780:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233358.769784:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233358.769787:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233358.769797:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895135488/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233358.769809:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233358/real 1761233358] req@00000000d67bb589 x1846780895135104/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233383 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233358.769840:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895135104/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233383 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233358.769856:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233358.769860:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233358.769867:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895135104:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233358.769878:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895135168:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233358.769914:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233358.769929:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233358/real 1761233358] req@000000009d1c7f73 x1846780895135168/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233383 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233358.769946:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895135168/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233383 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233358.769979:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233358.769983:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233358.769987:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895135168:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233358.770023:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895135232:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233358.770065:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233358.770078:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233358/real 1761233358] req@000000004130b6bd x1846780895135232/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233383 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233358.770097:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895135232/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233383 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233358.770108:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233358.770111:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233358.770116:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895135232:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233358.770125:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895135296:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233358.770155:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233358.770260:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233358/real 1761233358] req@00000000ef655201 x1846780895135296/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233383 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233358.770280:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895135296/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233383 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233358.770294:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233358.770298:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233358.770302:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895135296:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233358.770311:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895135360:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233358.770346:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233358.770359:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233358/real 1761233358] req@000000006d61dd06 x1846780895135360/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233383 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233358.770373:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895135360/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233383 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233358.770384:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233358.770387:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233358.770391:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895135360:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233358.770399:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895135424:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233358.770422:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233358.770433:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233358/real 1761233358] req@00000000efa29eb9 x1846780895135424/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233383 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233358.770449:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895135424/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233383 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233358.770460:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233358.770464:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233358.770468:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895135424:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233358.770476:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895135488:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233358.770503:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233358.770515:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233358/real 1761233358] req@000000001f9ec359 x1846780895135488/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233383 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233358.770531:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895135488/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233383 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233358.770541:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233358.770544:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233358.770549:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895135488:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233359.537443:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233360.561447:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233361.585497:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233362.609364:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233363.633536:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233363.889359:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233363.889369:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233363.889373:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233363.889384:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5924 00000100:00080000:0.0:1761233363.889387:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233363.889394:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233363.889428:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895135552/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233363.889454:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233363.889458:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233363.889459:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233363.889463:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5924 00000100:00080000:0.0:1761233363.889465:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233363.889469:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233363.889482:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233363.889496:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895135616/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233363.889507:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233363.889511:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233363.889513:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233363.889516:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895135552:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233363.889522:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5924 00000100:00080000:0.0:1761233363.889526:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233363.889530:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233363.889536:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233363.889550:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895135680/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233363.889557:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233363.889559:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233363.889560:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233363.889564:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5924 00000100:00080000:0.0:1761233363.889565:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233363.889568:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233363.889571:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233363.889578:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895135744/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233363.889584:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233363.889587:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233363.889590:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1761233363.889590:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233363.889596:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5924 00000100:00080000:0.0:1761233363.889598:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233363.889601:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233363.889605:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233363.889614:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895135808/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233363.889617:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233363/real 1761233363] req@000000005593c994 x1846780895135552/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233388 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233363.889636:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233363.889640:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233363.889642:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233363.889647:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895135552/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233388 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233363.889648:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5924 00000100:00080000:0.0:1761233363.889652:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233363.889657:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233363.889664:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761233363.889667:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233363.889671:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233363.889674:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895135872/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233363.889678:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895135552:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233363.889689:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233363.889691:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233363.889692:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233363.889696:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5924 00000100:00080000:0.0:1761233363.889697:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233363.889700:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233363.889702:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233363.889708:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895135616:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233363.889711:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895135936/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233363.889731:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233363.889738:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233363/real 1761233363] req@00000000c6e06344 x1846780895135616/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233388 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233363.889747:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895135616/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233388 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233363.889754:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233363.889756:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233363.889758:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895135616:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233363.889764:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895135680:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233363.889777:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233363.889785:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233363/real 1761233363] req@000000004862b808 x1846780895135680/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233388 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233363.889796:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895135680/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233388 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233363.889802:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233363.889804:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233363.889807:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895135680:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233363.889813:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895135744:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233363.889830:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233363.889842:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233363/real 1761233363] req@000000006f2efc8b x1846780895135744/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233388 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233363.889849:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895135744/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233388 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233363.889854:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233363.889856:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233363.889857:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895135744:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233363.889862:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895135808:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233363.889873:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233363.889877:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233363/real 1761233363] req@00000000eca9ddc2 x1846780895135808/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233388 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233363.889884:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895135808/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233388 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233363.889889:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233363.889890:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233363.889892:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895135808:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233363.889895:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895135872:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233363.889905:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233363.889909:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233363/real 1761233363] req@0000000034d49f59 x1846780895135872/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233388 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233363.889915:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895135872/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233388 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233363.889920:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233363.889921:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233363.889923:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895135872:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233363.889938:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895135936:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233363.889949:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233363.889953:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233363/real 1761233363] req@000000005beaae31 x1846780895135936/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233388 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233363.889960:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895135936/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233388 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233363.889965:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233363.889967:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233363.889969:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895135936:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233364.657449:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233365.681635:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233366.705498:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233367.729417:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233368.753338:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233369.009388:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233369.009400:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233369.009406:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233369.009419:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5929 00000100:00080000:2.0:1761233369.009423:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233369.009430:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233369.009473:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895136000/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233369.009500:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233369.009503:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233369.009505:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233369.009508:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5929 00000100:00080000:2.0:1761233369.009510:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233369.009514:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233369.009522:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233369.009537:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895136064/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233369.009562:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233369.009565:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233369.009568:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233369.009572:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5929 00000100:00080000:2.0:1761233369.009574:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233369.009577:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895136000:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761233369.009582:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233369.009588:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233369.009599:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895136128/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233369.009608:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233369.009612:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233369.009613:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233369.009617:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5929 00000100:00080000:2.0:1761233369.009619:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233369.009623:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233369.009627:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233369.009636:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895136192/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233369.009646:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233369.009650:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233369.009655:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233369.009657:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233369.009662:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5929 00000100:00080000:2.0:1761233369.009664:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233369.009670:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233369/real 1761233369] req@00000000ac7d8d5b x1846780895136000/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233394 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233369.009671:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233369.009678:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233369.009693:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895136256/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233369.009694:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895136000/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233394 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233369.009722:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233369.009723:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233369.009729:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761233369.009730:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1761233369.009733:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895136000:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761233369.009734:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233369.009743:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5929 00000100:00100000:3.0:1761233369.009747:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895136064:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:2.0:1761233369.009748:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233369.009755:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233369.009761:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233369.009776:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895136320/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233369.009777:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233369.009788:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233369.009792:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1761233369.009793:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233369/real 1761233369] req@000000004ae5bc9b x1846780895136064/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233394 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233369.009794:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233369.009817:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5929 00000100:00080000:2.0:1761233369.009821:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233369.009826:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233369.009831:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233369.009835:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895136064/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233394 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233369.009842:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895136384/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233369.009870:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233369.009874:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233369.009879:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895136064:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233369.009909:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895136128:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233369.009930:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233369.009937:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233369/real 1761233369] req@000000008ebccab5 x1846780895136128/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233394 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233369.009961:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895136128/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233394 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233369.009967:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233369.009968:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233369.009970:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895136128:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233369.009974:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895136192:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233369.009985:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233369.009990:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233369/real 1761233369] req@00000000a1e8c6bb x1846780895136192/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233394 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233369.009996:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895136192/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233394 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233369.010001:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233369.010002:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233369.010004:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895136192:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233369.010051:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895136256:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233369.010066:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233369.010071:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233369/real 1761233369] req@00000000c10cebe6 x1846780895136256/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233394 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233369.010078:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895136256/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233394 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233369.010083:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233369.010084:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233369.010086:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895136256:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233369.010089:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895136320:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233369.010099:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233369.010103:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233369/real 1761233369] req@0000000099b7e5d9 x1846780895136320/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233394 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233369.010110:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895136320/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233394 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233369.010114:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233369.010115:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233369.010117:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895136320:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233369.010120:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895136384:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233369.010129:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233369.010132:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233369/real 1761233369] req@00000000f4e415d3 x1846780895136384/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233394 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233369.010139:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895136384/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233394 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233369.010144:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233369.010145:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233369.010146:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895136384:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233369.777366:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233370.801394:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233371.825367:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233372.849378:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233373.873479:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233374.129387:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233374.129397:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233374.129400:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233374.129409:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5935 00000100:00080000:2.0:1761233374.129413:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233374.129419:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233374.129452:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895136448/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233374.129474:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233374.129476:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233374.129477:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233374.129480:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5935 00000100:00080000:2.0:1761233374.129481:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233374.129483:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233374.129489:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233374.129496:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895136512/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233374.129502:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233374.129504:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233374.129505:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233374.129507:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5935 00000100:00080000:2.0:1761233374.129508:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233374.129510:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233374.129512:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233374.129516:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895136576/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233374.129521:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233374.129522:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233374.129523:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233374.129525:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5935 00000100:00080000:2.0:1761233374.129527:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233374.129529:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233374.129532:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233374.129541:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895136640/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233374.129546:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233374.129548:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233374.129549:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233374.129551:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5935 00000100:00080000:2.0:1761233374.129554:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233374.129556:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233374.129559:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233374.129565:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895136704/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233374.129572:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233374.129575:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895136448:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761233374.129576:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233374.129579:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233374.129588:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5935 00000100:00080000:2.0:1761233374.129589:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233374.129591:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233374.129594:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233374.129605:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895136768/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233374.129614:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233374.129616:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233374.129617:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233374.129620:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5935 00000100:00080000:2.0:1761233374.129621:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233374.129622:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233374.129625:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233374.129632:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895136832/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233374.129669:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233374.129696:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233374/real 1761233374] req@00000000fd4aa3a2 x1846780895136448/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233399 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233374.129711:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895136448/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233399 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233374.129720:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233374.129723:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233374.129726:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895136448:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233374.129732:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895136512:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233374.129747:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233374.129751:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233374/real 1761233374] req@000000006c1041ba x1846780895136512/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233399 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233374.129758:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895136512/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233399 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233374.129763:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233374.129765:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233374.129767:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895136512:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233374.129771:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895136576:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233374.129782:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233374.129785:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233374/real 1761233374] req@000000006d8c3eb0 x1846780895136576/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233399 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233374.129792:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895136576/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233399 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233374.129796:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233374.129797:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233374.129799:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895136576:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233374.129802:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895136640:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233374.129812:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233374.129815:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233374/real 1761233374] req@0000000088939481 x1846780895136640/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233399 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233374.129821:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895136640/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233399 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233374.129825:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233374.129827:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233374.129828:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895136640:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233374.129857:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895136704:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233374.129868:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233374.129873:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233374/real 1761233374] req@000000006ff1fdbc x1846780895136704/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233399 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233374.129879:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895136704/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233399 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233374.129884:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233374.129886:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233374.129888:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895136704:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233374.129891:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895136768:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233374.129901:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233374.129904:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233374/real 1761233374] req@00000000135e9a6e x1846780895136768/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233399 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233374.129910:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895136768/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233399 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233374.129914:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233374.129915:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233374.129917:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895136768:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233374.129920:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895136832:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233374.129945:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233374.129950:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233374/real 1761233374] req@00000000fd3935a6 x1846780895136832/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233399 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233374.129957:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895136832/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233399 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233374.129962:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233374.129963:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233374.129965:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895136832:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233374.897360:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233375.921540:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233376.945453:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233377.969447:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233378.993441:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233379.249333:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233379.249344:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233379.249347:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233379.249360:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5940 00000100:00080000:0.0:1761233379.249363:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233379.249368:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233379.249400:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000078e7052d x1846780895136896/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233379.249425:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233379.249428:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233379.249429:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233379.249431:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5940 00000100:00080000:0.0:1761233379.249433:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233379.249435:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233379.249442:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233379.249454:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000addc3db8 x1846780895136960/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233379.249460:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233379.249462:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233379.249463:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233379.249465:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5940 00000100:00080000:0.0:1761233379.249467:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233379.249469:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233379.249474:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233379.249481:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895137024/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233379.249487:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233379.249489:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233379.249490:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233379.249492:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5940 00000100:00080000:0.0:1761233379.249493:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233379.249495:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233379.249498:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233379.249499:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895136896:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233379.249505:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895137088/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233379.249511:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233379.249513:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233379.249514:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233379.249516:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5940 00000100:00080000:0.0:1761233379.249518:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233379.249520:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233379.249522:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233379.249528:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895137152/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233379.249533:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233379.249535:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233379.249536:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233379.249540:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5940 00000100:00080000:0.0:1761233379.249541:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233379.249544:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233379.249547:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233379.249554:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895137216/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233379.249559:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233379.249562:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233379.249563:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233379.249565:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5940 00000100:00080000:0.0:1761233379.249566:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233379.249569:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233379.249572:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233379.249576:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233379.249580:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895137280/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233379.249605:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233379/real 1761233379] req@0000000078e7052d x1846780895136896/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233404 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233379.249623:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000078e7052d x1846780895136896/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233404 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233379.249633:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233379.249636:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233379.249640:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895136896:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233379.249654:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895136960:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233379.249679:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233379.249688:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233379/real 1761233379] req@00000000addc3db8 x1846780895136960/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233404 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233379.249699:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000addc3db8 x1846780895136960/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233404 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233379.249713:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233379.249716:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233379.249719:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895136960:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233379.249725:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895137024:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233379.249747:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233379.249755:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233379/real 1761233379] req@000000005beaae31 x1846780895137024/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233404 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233379.249767:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895137024/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233404 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233379.249774:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233379.249776:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233379.249779:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895137024:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233379.249806:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895137088:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233379.249827:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233379.249835:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233379/real 1761233379] req@0000000034d49f59 x1846780895137088/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233404 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233379.249865:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895137088/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233404 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233379.249873:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233379.249887:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233379.249890:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895137088:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233379.249896:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895137152:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233379.249918:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233379.249925:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233379/real 1761233379] req@00000000eca9ddc2 x1846780895137152/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233404 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233379.249935:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895137152/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233404 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233379.249942:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233379.249944:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233379.249947:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895137152:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233379.249953:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895137216:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233379.249973:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233379.249992:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233379/real 1761233379] req@000000006f2efc8b x1846780895137216/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233404 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233379.250003:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895137216/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233404 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233379.250011:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233379.250013:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233379.250016:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895137216:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233379.250022:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895137280:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233379.250041:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233379.250049:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233379/real 1761233379] req@000000004862b808 x1846780895137280/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233404 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233379.250061:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895137280/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233404 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233379.250068:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233379.250070:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233379.250073:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895137280:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233380.017480:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233381.041421:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233382.065414:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233383.089457:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233384.113370:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233384.369306:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233384.369315:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233384.369319:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233384.369329:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5945 00000100:00080000:0.0:1761233384.369332:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233384.369338:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233384.369372:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895137344/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233384.369395:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233384.369398:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233384.369399:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233384.369402:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5945 00000100:00080000:0.0:1761233384.369403:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233384.369405:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233384.369412:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233384.369419:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895137408/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233384.369424:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233384.369426:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233384.369427:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233384.369430:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5945 00000100:00080000:0.0:1761233384.369431:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233384.369433:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233384.369436:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233384.369442:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895137472/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233384.369446:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233384.369449:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233384.369449:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233384.369451:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5945 00000100:00080000:0.0:1761233384.369453:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233384.369455:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233384.369458:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233384.369465:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895137344:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233384.369466:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895137536/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233384.369478:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233384.369480:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233384.369482:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233384.369484:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5945 00000100:00080000:0.0:1761233384.369485:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233384.369488:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233384.369490:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233384.369498:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895137600/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233384.369502:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233384.369505:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233384.369506:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233384.369508:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5945 00000100:00080000:0.0:1761233384.369509:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233384.369511:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233384.369514:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233384.369520:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895137664/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233384.369524:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233384.369525:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233384.369526:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233384.369528:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5945 00000100:00080000:0.0:1761233384.369530:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233384.369532:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233384.369535:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233384.369545:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895137728/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233384.369545:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233384.369572:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233384/real 1761233384] req@00000000c6e06344 x1846780895137344/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233409 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233384.369587:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895137344/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233409 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233384.369597:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233384.369600:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233384.369604:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895137344:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233384.369611:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895137408:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233384.369634:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233384.369642:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233384/real 1761233384] req@000000005593c994 x1846780895137408/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233409 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233384.369654:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895137408/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233409 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233384.369663:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233384.369666:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233384.369669:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895137408:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233384.369684:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895137472:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233384.369704:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233384.369711:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233384/real 1761233384] req@000000001f9ec359 x1846780895137472/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233409 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233384.369722:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895137472/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233409 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233384.369730:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233384.369732:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233384.369735:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895137472:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233384.369764:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895137536:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233384.369829:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233384.369838:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233384/real 1761233384] req@00000000efa29eb9 x1846780895137536/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233409 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233384.369850:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895137536/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233409 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233384.369868:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233384.369871:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233384.369874:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895137536:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233384.369881:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895137600:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233384.369898:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233384.369906:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233384/real 1761233384] req@000000006d61dd06 x1846780895137600/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233409 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233384.369917:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895137600/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233409 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233384.369924:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233384.369927:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233384.369930:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895137600:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233384.369936:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895137664:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233384.369952:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233384.369959:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233384/real 1761233384] req@00000000ef655201 x1846780895137664/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233409 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233384.369970:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895137664/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233409 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233384.369978:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233384.369980:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233384.369983:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895137664:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233384.369989:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895137728:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233384.370006:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233384.370013:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233384/real 1761233384] req@000000004130b6bd x1846780895137728/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233409 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233384.370036:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895137728/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233409 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233384.370044:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233384.370046:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233384.370049:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895137728:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233385.137454:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233386.161363:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233387.185365:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233388.209413:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233389.233423:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233389.489333:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233389.489346:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233389.489350:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233389.489362:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5950 00000100:00080000:0.0:1761233389.489366:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233389.489373:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233389.489412:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895137792/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233389.489441:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233389.489444:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233389.489445:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233389.489448:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5950 00000100:00080000:0.0:1761233389.489449:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233389.489452:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233389.489459:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233389.489468:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895137856/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233389.489474:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233389.489476:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233389.489490:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233389.489492:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5950 00000100:00080000:0.0:1761233389.489494:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233389.489496:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233389.489499:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233389.489504:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895137920/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233389.489509:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233389.489511:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233389.489512:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233389.489515:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5950 00000100:00080000:0.0:1761233389.489516:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233389.489518:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233389.489521:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233389.489525:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895137792:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233389.489537:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895137984/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233389.489544:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233389.489546:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233389.489548:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233389.489550:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5950 00000100:00080000:0.0:1761233389.489552:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233389.489555:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233389.489558:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233389.489567:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895138048/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233389.489573:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233389.489576:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233389.489577:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233389.489579:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5950 00000100:00080000:0.0:1761233389.489580:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233389.489582:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233389.489585:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233389.489592:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895138112/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233389.489598:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1761233389.489601:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233389.489602:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233389.489605:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233389.489609:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5950 00000100:00080000:0.0:1761233389.489610:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233389.489612:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233389.489616:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233389.489623:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895138176/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233389.489633:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233389/real 1761233389] req@000000009d1c7f73 x1846780895137792/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233414 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233389.489664:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895137792/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233414 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233389.489674:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233389.489677:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233389.489681:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895137792:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233389.489720:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895137856:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233389.489746:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233389.489757:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233389/real 1761233389] req@00000000d67bb589 x1846780895137856/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233414 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233389.489768:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895137856/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233414 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233389.489781:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233389.489784:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233389.489787:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895137856:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233389.489793:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895137920:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233389.489812:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233389.489819:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233389/real 1761233389] req@00000000a4a2c44a x1846780895137920/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233414 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233389.489829:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895137920/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233414 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233389.489836:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233389.489838:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233389.489841:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895137920:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233389.489869:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895137984:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233389.489890:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233389.489898:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233389/real 1761233389] req@000000004a74da19 x1846780895137984/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233414 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233389.489909:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895137984/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233414 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233389.489916:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233389.489918:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233389.489921:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895137984:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233389.489926:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895138048:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233389.489944:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233389.489952:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233389/real 1761233389] req@000000002d1b8ef7 x1846780895138048/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233414 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233389.489961:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895138048/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233414 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233389.489967:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233389.489969:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233389.489972:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895138048:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233389.489977:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895138112:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233389.489993:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233389.490001:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233389/real 1761233389] req@000000007bd1ec64 x1846780895138112/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233414 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233389.490011:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895138112/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233414 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233389.490017:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233389.490019:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233389.490021:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895138112:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233389.490027:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895138176:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233389.490044:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233389.490051:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233389/real 1761233389] req@000000001db96463 x1846780895138176/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233414 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233389.490061:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895138176/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233414 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233389.490067:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233389.490069:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233389.490072:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895138176:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233390.257376:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233391.281384:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233392.305388:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233393.329393:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233394.353337:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233394.609296:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233394.609309:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233394.609313:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233394.609324:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5955 00000100:00080000:0.0:1761233394.609328:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233394.609336:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233394.609373:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895138240/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233394.609400:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233394.609404:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233394.609405:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233394.609407:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5955 00000100:00080000:0.0:1761233394.609409:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233394.609412:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233394.609420:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233394.609429:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895138304/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233394.609435:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233394.609438:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233394.609440:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233394.609443:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5955 00000100:00080000:0.0:1761233394.609445:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233394.609447:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233394.609451:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233394.609458:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895138368/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233394.609465:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233394.609470:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1761233394.609470:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895138240:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233394.609473:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233394.609480:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5955 00000100:00080000:0.0:1761233394.609482:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233394.609485:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233394.609488:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233394.609497:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895138432/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233394.609502:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233394.609505:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233394.609506:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233394.609509:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5955 00000100:00080000:0.0:1761233394.609511:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233394.609514:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233394.609517:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233394.609523:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895138496/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233394.609529:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233394.609532:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233394.609533:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233394.609535:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5955 00000100:00080000:0.0:1761233394.609536:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233394.609550:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233394.609555:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233394.609555:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233394.609566:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895138560/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233394.609577:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233394.609580:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233394.609581:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233394.609584:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5955 00000100:00080000:0.0:1761233394.609585:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233394.609588:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233394.609590:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233394.609598:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233394/real 1761233394] req@00000000f3ed2588 x1846780895138240/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233419 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233394.609600:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895138624/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233394.609646:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f3ed2588 x1846780895138240/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233419 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233394.609658:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233394.609661:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233394.609666:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895138240:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233394.609673:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895138304:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233394.609691:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233394.609697:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233394/real 1761233394] req@000000005d83493f x1846780895138304/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233419 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233394.609705:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005d83493f x1846780895138304/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233419 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233394.609711:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233394.609713:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233394.609715:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895138304:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233394.609737:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895138368:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233394.609750:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233394.609755:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233394/real 1761233394] req@000000001db96463 x1846780895138368/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233419 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233394.609762:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895138368/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233419 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233394.609767:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233394.609769:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233394.609771:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895138368:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233394.609775:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895138432:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233394.609785:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233394.609789:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233394/real 1761233394] req@000000007bd1ec64 x1846780895138432/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233419 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233394.609796:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895138432/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233419 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233394.609801:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233394.609803:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233394.609805:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895138432:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233394.609809:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895138496:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233394.609820:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233394.609825:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233394/real 1761233394] req@000000002d1b8ef7 x1846780895138496/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233419 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233394.609832:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895138496/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233419 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233394.609837:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233394.609838:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233394.609840:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895138496:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233394.609843:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895138560:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233394.609854:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233394.609859:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233394/real 1761233394] req@000000004a74da19 x1846780895138560/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233419 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233394.609865:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895138560/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233419 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233394.609879:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233394.609880:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233394.609882:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895138560:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233394.609885:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895138624:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233394.609895:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233394.609900:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233394/real 1761233394] req@00000000a4a2c44a x1846780895138624/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233419 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233394.609908:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895138624/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233419 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233394.609913:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233394.609914:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233394.609916:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895138624:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233395.377398:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233396.401396:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233397.425446:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233398.449407:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233399.473404:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233399.729450:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233399.729464:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233399.729469:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233399.729483:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5960 00000100:00080000:1.0:1761233399.729488:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233399.729496:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233399.729557:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac65ecb7 x1846780895138688/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233399.729588:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233399.729592:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233399.729594:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233399.729598:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5960 00000100:00080000:1.0:1761233399.729601:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233399.729604:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233399.729615:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233399.729625:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006a5e9865 x1846780895138752/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233399.729633:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233399.729635:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233399.729636:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233399.729640:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5960 00000100:00080000:1.0:1761233399.729642:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233399.729644:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233399.729647:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233399.729654:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004361f727 x1846780895138816/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233399.729663:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1761233399.729665:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac65ecb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895138688:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233399.729668:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233399.729670:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233399.729682:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5960 00000100:00080000:1.0:1761233399.729685:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233399.729688:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233399.729692:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233399.729707:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df0b94a8 x1846780895138880/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233399.729713:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233399.729716:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233399.729718:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233399.729722:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5960 00000100:00080000:1.0:1761233399.729725:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233399.729727:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233399.729730:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233399.729739:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df5b2270 x1846780895138944/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233399.729743:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233399.729745:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233399.729746:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233399.729748:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5960 00000100:00080000:1.0:1761233399.729750:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233399.729753:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233399.729757:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1761233399.729765:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233399.729770:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ffe86d04 x1846780895139008/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233399.729779:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233399.729781:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233399.729784:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1761233399.729811:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233399/real 1761233399] req@00000000ac65ecb7 x1846780895138688/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233424 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233399.729813:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5960 00000100:00080000:1.0:1761233399.729833:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233399.729842:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233399.729852:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233399.729905:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d66930c2 x1846780895139072/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233399.729906:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac65ecb7 x1846780895138688/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233424 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233399.729996:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233399.730005:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233399.730013:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac65ecb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895138688:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761233399.730025:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a5e9865 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895138752:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761233399.730056:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233399.730066:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233399/real 1761233399] req@000000006a5e9865 x1846780895138752/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233424 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233399.730088:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006a5e9865 x1846780895138752/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233424 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233399.730097:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233399.730100:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233399.730103:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a5e9865 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895138752:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233399.730137:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004361f727 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895138816:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233399.730167:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233399.730182:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233399/real 1761233399] req@000000004361f727 x1846780895138816/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233424 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233399.730202:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004361f727 x1846780895138816/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233424 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233399.730215:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233399.730220:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233399.730226:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004361f727 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895138816:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233399.730282:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895138880:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233399.730339:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233399.730351:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233399/real 1761233399] req@00000000df0b94a8 x1846780895138880/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233424 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233399.730363:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df0b94a8 x1846780895138880/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233424 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233399.730371:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233399.730374:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233399.730377:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895138880:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233399.730383:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895138944:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761233399.730403:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233399.730410:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233399/real 1761233399] req@00000000df5b2270 x1846780895138944/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233424 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233399.730438:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df5b2270 x1846780895138944/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233424 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233399.730447:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233399.730449:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233399.730452:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895138944:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233399.730459:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895139008:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233399.730479:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233399.730487:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233399/real 1761233399] req@00000000ffe86d04 x1846780895139008/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233424 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233399.730515:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ffe86d04 x1846780895139008/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233424 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233399.730549:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233399.730555:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233399.730560:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895139008:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233399.730572:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895139072:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233399.730614:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233399.730631:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233399/real 1761233399] req@00000000d66930c2 x1846780895139072/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233424 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233399.730653:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d66930c2 x1846780895139072/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233424 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233399.730666:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233399.730670:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233399.730675:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895139072:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233400.497464:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233401.521395:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233402.545452:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233403.569430:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233404.593463:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233404.849363:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233404.849378:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233404.849384:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233404.849399:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5965 00000100:00080000:1.0:1761233404.849404:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233404.849412:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233404.849500:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090ac513e x1846780895139136/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233404.849531:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233404.849535:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233404.849537:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233404.849540:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5965 00000100:00080000:1.0:1761233404.849542:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233404.849545:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233404.849556:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233404.849566:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000053e52331 x1846780895139200/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233404.849575:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233404.849579:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233404.849581:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233404.849586:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5965 00000100:00080000:1.0:1761233404.849590:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233404.849595:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233404.849601:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233404.849609:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000880f16b4 x1846780895139264/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233404.849617:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233404.849620:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233404.849623:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1761233404.849629:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895139136:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233404.849631:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5965 00000100:00080000:1.0:1761233404.849636:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233404.849644:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233404.849650:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233404.849665:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cb5be845 x1846780895139328/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233404.849672:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233404.849675:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233404.849677:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233404.849680:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5965 00000100:00080000:1.0:1761233404.849682:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233404.849686:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233404.849693:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233404.849716:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a95774fb x1846780895139392/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233404.849727:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233404.849732:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233404.849735:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1761233404.849737:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:1.0:1761233404.849750:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5965 00000100:00080000:1.0:1761233404.849754:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233404.849760:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233404.849767:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233404.849773:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233404/real 1761233404] req@0000000090ac513e x1846780895139136/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233429 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233404.849788:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000c736bb2 x1846780895139456/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233404.849802:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090ac513e x1846780895139136/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233429 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233404.849826:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233404.849827:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233404.849831:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:1.0:1761233404.849838:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1761233404.849841:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895139136:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233404.849844:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1761233404.849851:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895139200:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:1.0:1761233404.849858:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5965 00000100:00080000:1.0:1761233404.849866:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233404.849874:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233404.849882:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1761233404.849886:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233404.849897:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233404/real 1761233404] req@0000000053e52331 x1846780895139200/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233429 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233404.849910:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000038209f5 x1846780895139520/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233404.849912:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000053e52331 x1846780895139200/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233429 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233404.849968:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233404.849974:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233404.849980:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895139200:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233404.850019:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895139264:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233404.850038:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233404.850044:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233404/real 1761233404] req@00000000880f16b4 x1846780895139264/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233429 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233404.850053:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000880f16b4 x1846780895139264/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233429 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233404.850060:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233404.850062:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233404.850064:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895139264:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233404.850069:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895139328:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233404.850083:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233404.850088:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233404/real 1761233404] req@00000000cb5be845 x1846780895139328/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233429 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233404.850097:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cb5be845 x1846780895139328/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233429 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233404.850118:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233404.850120:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233404.850123:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895139328:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233404.850129:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895139392:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761233404.850146:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233404.850152:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233404/real 1761233404] req@00000000a95774fb x1846780895139392/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233429 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233404.850161:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a95774fb x1846780895139392/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233429 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233404.850168:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233404.850170:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233404.850172:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895139392:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233404.850177:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895139456:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233404.850269:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233404.850277:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233404/real 1761233404] req@000000000c736bb2 x1846780895139456/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233429 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233404.850285:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000000c736bb2 x1846780895139456/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233429 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233404.850292:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233404.850294:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233404.850297:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895139456:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233404.850301:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895139520:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233404.850316:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233404.850321:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233404/real 1761233404] req@00000000038209f5 x1846780895139520/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233429 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233404.850330:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000038209f5 x1846780895139520/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233429 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233404.850336:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233404.850338:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233404.850340:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895139520:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233405.617430:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233406.641436:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233407.665348:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233408.689449:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233409.713393:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233409.969314:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233409.969323:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233409.969326:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233409.969335:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5970 00000100:00080000:1.0:1761233409.969338:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233409.969345:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233409.969410:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a611ee34 x1846780895139584/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233409.969439:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233409.969443:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233409.969445:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233409.969449:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5970 00000100:00080000:1.0:1761233409.969451:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233409.969454:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233409.969463:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233409.969475:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000328368c8 x1846780895139648/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233409.969482:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233409.969484:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233409.969485:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233409.969488:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5970 00000100:00080000:1.0:1761233409.969489:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1761233409.969494:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895139584:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:1.0:1761233409.969494:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233409.969498:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233409.969505:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c652d0ad x1846780895139712/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233409.969511:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233409.969513:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233409.969514:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233409.969517:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5970 00000100:00080000:1.0:1761233409.969519:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233409.969521:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233409.969523:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233409.969533:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ba3340f x1846780895139776/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233409.969538:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233409.969540:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233409.969541:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233409.969543:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5970 00000100:00080000:1.0:1761233409.969544:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233409.969547:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233409.969549:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233409.969556:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d74b2e3c x1846780895139840/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233409.969561:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233409.969563:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233409.969564:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233409.969566:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5970 00000100:00080000:1.0:1761233409.969567:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233409.969570:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233409.969573:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233409.969580:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001aedf9b1 x1846780895139904/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233409.969581:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895139648:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:1.0:1761233409.969595:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233409.969598:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233409.969599:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233409.969602:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5970 00000100:00080000:1.0:1761233409.969603:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233409.969606:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233409.969609:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233409.969618:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895139712:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233409.969619:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000568f2a96 x1846780895139968/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233409.969641:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895139776:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233409.969655:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895139840:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761233409.969668:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895139904:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233409.969682:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895139968:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233410.737424:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233411.761471:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233412.785473:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1761233413.041645:0:2209:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:0.0:1761233413.041696:0:2209:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1761233413.041712:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233413.041763:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233413.041772:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233413.041781:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233413.041789:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233413.041798:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233413.041806:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:3.0:1761233413.041854:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233409/real 1761233413] req@00000000568f2a96 x1846780895139968/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233434 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233413.041881:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000568f2a96 x1846780895139968/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233434 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233413.041893:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233413.041898:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233413.041904:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895139968:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233413.041911:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233409/real 1761233413] req@00000000c652d0ad x1846780895139712/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233434 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233413.041928:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c652d0ad x1846780895139712/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233434 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233413.041936:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233413.041939:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233413.041942:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895139712:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233413.041947:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233409/real 1761233413] req@000000007ba3340f x1846780895139776/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233434 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233413.041957:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007ba3340f x1846780895139776/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233434 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233413.041964:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233413.041967:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233413.041969:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895139776:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233413.041975:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233409/real 1761233413] req@00000000d74b2e3c x1846780895139840/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233434 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233413.041984:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d74b2e3c x1846780895139840/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233434 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233413.041991:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233413.041994:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233413.041996:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895139840:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233413.042001:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233409/real 1761233413] req@000000001aedf9b1 x1846780895139904/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233434 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233413.042017:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001aedf9b1 x1846780895139904/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233434 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233413.042023:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233413.042025:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233413.042028:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895139904:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233413.042033:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233409/real 1761233413] req@00000000a611ee34 x1846780895139584/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233434 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233413.042043:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a611ee34 x1846780895139584/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233434 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233413.042049:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233413.042052:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233413.042055:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895139584:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233413.042060:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233409/real 1761233413] req@00000000328368c8 x1846780895139648/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233434 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233413.042069:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000328368c8 x1846780895139648/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233434 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233413.042075:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233413.042077:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233413.042080:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895139648:192.168.201.157@tcp:38:kworker/u8:2.0 00010000:00010000:1.0:1761233413.809446:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233414.833425:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233415.089357:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233415.089370:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233415.089374:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233415.089386:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5976 00000100:00080000:0.0:1761233415.089389:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233415.089396:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233415.089428:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895140032/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233415.089456:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233415.089459:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233415.089461:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233415.089464:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5976 00000100:00080000:0.0:1761233415.089466:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233415.089469:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233415.089478:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233415.089489:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895140096/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233415.089499:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233415.089503:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1761233415.089504:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895140032:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233415.089505:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233415.089514:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5976 00000100:00080000:0.0:1761233415.089517:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233415.089520:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233415.089526:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233415.089538:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895140160/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233415.089545:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233415.089549:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233415.089550:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233415.089554:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5976 00000100:00080000:0.0:1761233415.089556:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233415.089559:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233415.089564:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233415.089573:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895140224/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233415.089581:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233415.089586:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233415.089590:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233415.089592:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233415.089596:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5976 00000100:00080000:0.0:1761233415.089599:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233415.089603:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233415.089606:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233415/real 1761233415] req@00000000d67bb589 x1846780895140032/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233440 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:0.0:1761233415.089609:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233415.089623:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895140288/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233415.089629:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895140032/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233440 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233415.089635:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233415.089643:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233415.089647:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233415.089650:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233415.089654:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5976 00000100:00080000:3.0:1761233415.089656:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233415.089658:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233415.089664:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895140032:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233415.089666:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233415.089674:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233415.089687:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895140352/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233415.089696:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233415.089696:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895140096:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:0.0:1761233415.089703:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233415.089706:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233415.089711:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5976 00000100:00080000:0.0:1761233415.089714:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233415.089718:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233415.089723:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233415.089730:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233415.089736:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895140416/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233415.089744:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233415/real 1761233415] req@000000009d1c7f73 x1846780895140096/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233440 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233415.089759:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895140096/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233440 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233415.089768:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233415.089772:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233415.089776:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895140096:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233415.089783:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895140160:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233415.089803:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233415.089811:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233415/real 1761233415] req@000000004130b6bd x1846780895140160/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233440 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233415.089822:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895140160/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233440 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233415.089840:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233415.089843:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233415.089846:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895140160:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233415.089851:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895140224:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233415.089872:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233415.089880:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233415/real 1761233415] req@00000000ef655201 x1846780895140224/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233440 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233415.089891:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895140224/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233440 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233415.089897:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233415.089899:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233415.089902:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895140224:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233415.089908:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895140288:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233415.089926:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233415.089934:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233415/real 1761233415] req@000000006d61dd06 x1846780895140288/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233440 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233415.089944:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895140288/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233440 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233415.089951:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233415.089953:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233415.089956:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895140288:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233415.089976:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895140352:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233415.089996:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233415.090005:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233415/real 1761233415] req@00000000efa29eb9 x1846780895140352/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233440 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233415.090015:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895140352/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233440 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233415.090021:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233415.090023:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233415.090026:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895140352:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233415.090031:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895140416:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233415.090049:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233415.090057:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233415/real 1761233415] req@000000001f9ec359 x1846780895140416/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233440 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233415.090066:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895140416/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233440 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233415.090073:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233415.090074:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233415.090077:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895140416:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233415.857494:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233416.881418:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233417.905424:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233418.929430:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233419.953399:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233420.209349:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233420.209362:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233420.209382:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233420.209391:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5981 00000100:00080000:1.0:1761233420.209395:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233420.209400:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233420.209431:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cbad49de x1846780895140480/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233420.209452:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233420.209455:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233420.209456:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233420.209459:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5981 00000100:00080000:1.0:1761233420.209460:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233420.209464:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233420.209472:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233420.209484:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e467e107 x1846780895140544/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233420.209485:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbad49de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895140480:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:1.0:1761233420.209499:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233420.209501:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233420.209501:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233420.209504:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5981 00000100:00080000:1.0:1761233420.209505:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233420.209507:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233420.209511:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233420.209522:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090800dde x1846780895140608/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233420.209531:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233420.209535:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233420.209536:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233420.209539:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5981 00000100:00080000:1.0:1761233420.209542:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233420.209546:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233420.209550:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233420.209559:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233420.209563:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000328368c8 x1846780895140672/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233420.209574:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233420.209578:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233420.209581:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233420.209585:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5981 00000100:00080000:1.0:1761233420.209589:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233420.209589:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233420/real 1761233420] req@00000000cbad49de x1846780895140480/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233445 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233420.209597:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233420.209605:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233420.209609:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cbad49de x1846780895140480/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233445 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233420.209618:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a611ee34 x1846780895140736/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233420.209619:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233420.209624:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233420.209631:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbad49de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895140480:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:1.0:1761233420.209633:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233420.209638:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233420.209639:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233420.209643:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5981 00000100:00080000:1.0:1761233420.209644:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233420.209648:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233420.209651:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233420.209658:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e467e107 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895140544:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233420.209660:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001aedf9b1 x1846780895140800/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233420.209671:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233420.209673:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233420.209674:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233420.209677:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5981 00000100:00080000:1.0:1761233420.209678:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233420.209681:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233420.209684:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233420.209688:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233420.209695:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d74b2e3c x1846780895140864/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233420.209699:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233420/real 1761233420] req@00000000e467e107 x1846780895140544/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233445 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233420.209721:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e467e107 x1846780895140544/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233445 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233420.209727:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233420.209730:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233420.209733:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e467e107 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895140544:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233420.209738:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090800dde pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895140608:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233420.209752:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233420.209756:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233420/real 1761233420] req@0000000090800dde x1846780895140608/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233445 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233420.209768:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090800dde x1846780895140608/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233445 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233420.209772:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233420.209773:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233420.209775:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090800dde pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895140608:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233420.209778:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895140672:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233420.209788:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233420.209792:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233420/real 1761233420] req@00000000328368c8 x1846780895140672/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233445 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233420.209798:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000328368c8 x1846780895140672/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233445 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233420.209803:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233420.209804:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233420.209806:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895140672:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233420.209809:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895140736:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233420.209819:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233420.209823:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233420/real 1761233420] req@00000000a611ee34 x1846780895140736/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233445 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233420.209829:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a611ee34 x1846780895140736/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233445 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233420.209834:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233420.209836:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233420.209838:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895140736:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233420.209850:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895140800:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233420.209860:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233420.209864:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233420/real 1761233420] req@000000001aedf9b1 x1846780895140800/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233445 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233420.209870:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001aedf9b1 x1846780895140800/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233445 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233420.209875:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233420.209876:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233420.209878:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895140800:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233420.209881:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895140864:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233420.209890:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233420.209894:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233420/real 1761233420] req@00000000d74b2e3c x1846780895140864/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233445 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233420.209901:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d74b2e3c x1846780895140864/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233445 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233420.209906:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233420.209907:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233420.209909:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895140864:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233420.977433:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233422.001390:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233423.025366:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233424.049403:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233425.073348:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233425.329347:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233425.329359:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233425.329364:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233425.329376:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5986 00000100:00080000:0.0:1761233425.329381:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233425.329389:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233425.329435:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895140928/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233425.329466:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233425.329470:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233425.329472:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233425.329475:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5986 00000100:00080000:0.0:1761233425.329477:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233425.329481:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233425.329491:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233425.329501:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895140992/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233425.329510:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233425.329513:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233425.329515:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233425.329519:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5986 00000100:00080000:0.0:1761233425.329520:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233425.329524:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233425.329528:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233425.329537:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895140928:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233425.329539:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895141056/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233425.329554:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233425.329557:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233425.329558:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233425.329561:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5986 00000100:00080000:0.0:1761233425.329562:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233425.329565:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233425.329569:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233425.329579:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895141120/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233425.329586:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233425.329589:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233425.329590:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233425.329593:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5986 00000100:00080000:0.0:1761233425.329595:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233425.329598:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233425.329601:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233425.329611:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895141184/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233425.329620:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233425.329625:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233425.329629:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233425.329631:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233425.329636:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5986 00000100:00080000:0.0:1761233425.329638:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233425.329642:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233425.329646:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233425.329649:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233425/real 1761233425] req@000000005593c994 x1846780895140928/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233450 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233425.329658:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895141248/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233425.329674:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233425.329677:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233425.329679:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233425.329681:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895140928/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233450 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233425.329687:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5986 00000100:00080000:0.0:1761233425.329691:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233425.329697:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:3.0:1761233425.329719:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1761233425.329720:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761233425.329727:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233425.329736:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895140928:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233425.329740:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895141312/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233425.329752:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895140992:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233425.329784:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233425.329793:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233425/real 1761233425] req@00000000c6e06344 x1846780895140992/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233450 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233425.329804:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895140992/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233450 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233425.329812:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233425.329814:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233425.329817:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895140992:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233425.329847:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895141056:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233425.329864:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233425.329871:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233425/real 1761233425] req@000000004862b808 x1846780895141056/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233450 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233425.329882:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895141056/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233450 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233425.329905:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233425.329907:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233425.329909:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895141056:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233425.329914:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895141120:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233425.329927:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233425.329933:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233425/real 1761233425] req@000000006f2efc8b x1846780895141120/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233450 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233425.329941:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895141120/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233450 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233425.329949:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233425.329952:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233425.329956:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895141120:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233425.329964:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895141184:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233425.329990:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233425.330001:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233425/real 1761233425] req@00000000eca9ddc2 x1846780895141184/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233450 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233425.330016:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895141184/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233450 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233425.330026:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233425.330030:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233425.330034:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895141184:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233425.330042:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895141248:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233425.330062:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233425.330069:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233425/real 1761233425] req@0000000034d49f59 x1846780895141248/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233450 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233425.330078:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895141248/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233450 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233425.330084:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233425.330086:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233425.330088:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895141248:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233425.330093:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895141312:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233425.330105:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233425.330110:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233425/real 1761233425] req@000000005beaae31 x1846780895141312/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233450 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233425.330118:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895141312/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233450 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233425.330146:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233425.330148:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233425.330151:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895141312:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233426.097380:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233427.121381:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233428.145404:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233429.169365:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233430.193438:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233430.449328:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233430.449340:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233430.449344:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233430.449355:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5991 00000100:00080000:0.0:1761233430.449359:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233430.449367:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233430.449399:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000addc3db8 x1846780895141376/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233430.449424:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233430.449428:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233430.449430:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233430.449433:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5991 00000100:00080000:0.0:1761233430.449435:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233430.449439:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233430.449448:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233430.449460:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000078e7052d x1846780895141440/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233430.449461:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895141376:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233430.449471:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233430.449474:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233430.449476:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233430.449480:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5991 00000100:00080000:0.0:1761233430.449482:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233430.449484:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233430.449488:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233430.449496:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895141504/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233430.449504:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233430.449506:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233430.449507:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233430.449509:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5991 00000100:00080000:0.0:1761233430.449511:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233430.449513:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233430.449516:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233430.449523:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895141568/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761233430.449525:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233430.449530:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233430.449532:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233430.449534:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233430.449536:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5991 00000100:00080000:0.0:1761233430.449538:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233430.449541:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233430.449544:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233430.449546:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233430/real 1761233430] req@00000000addc3db8 x1846780895141376/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233455 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233430.449551:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895141632/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233430.449559:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233430.449561:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000addc3db8 x1846780895141376/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233455 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233430.449562:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233430.449563:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233430.449567:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5991 00000100:00080000:0.0:1761233430.449568:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233430.449571:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:1.0:1761233430.449571:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233430.449574:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000008:00100000:0.0:1761233430.449575:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233430.449577:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895141376:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233430.449583:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895141696/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233430.449589:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233430.449592:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233430.449593:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233430.449596:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5991 00000100:00080000:0.0:1761233430.449598:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761233430.449598:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895141440:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:0.0:1761233430.449601:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233430.449604:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233430.449611:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895141760/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761233430.449618:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233430.449625:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233430/real 1761233430] req@0000000078e7052d x1846780895141440/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233455 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233430.449636:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000078e7052d x1846780895141440/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233455 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233430.449643:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233430.449645:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233430.449648:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895141440:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233430.449653:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895141504:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233430.449673:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233430.449679:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233430/real 1761233430] req@000000005beaae31 x1846780895141504/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233455 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233430.449685:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895141504/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233455 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233430.449690:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233430.449691:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233430.449693:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895141504:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233430.449696:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895141568:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233430.449706:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233430.449710:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233430/real 1761233430] req@0000000034d49f59 x1846780895141568/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233455 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233430.449715:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895141568/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233455 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233430.449720:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233430.449721:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233430.449722:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895141568:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233430.449725:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895141632:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233430.449735:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233430.449738:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233430/real 1761233430] req@00000000eca9ddc2 x1846780895141632/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233455 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233430.449744:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895141632/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233455 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233430.449749:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233430.449750:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233430.449752:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895141632:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233430.449754:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895141696:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233430.449764:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233430.449767:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233430/real 1761233430] req@000000006f2efc8b x1846780895141696/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233455 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233430.449773:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895141696/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233455 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233430.449777:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233430.449778:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233430.449780:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895141696:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233430.449793:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895141760:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233430.449802:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233430.449806:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233430/real 1761233430] req@000000004862b808 x1846780895141760/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233455 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233430.449812:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895141760/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233455 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233430.449816:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233430.449818:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233430.449819:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895141760:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233431.217366:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233432.241464:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233433.265351:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233434.289464:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233435.313523:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233435.569384:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233435.569393:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233435.569396:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233435.569407:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 5996 00000100:00080000:1.0:1761233435.569410:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233435.569416:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233435.569449:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ba3340f x1846780895141824/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233435.569478:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233435.569481:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233435.569482:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233435.569485:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5996 00000100:00080000:1.0:1761233435.569486:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233435.569489:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233435.569496:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233435.569504:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c652d0ad x1846780895141888/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233435.569510:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233435.569512:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233435.569513:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233435.569521:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5996 00000100:00080000:1.0:1761233435.569523:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233435.569525:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233435.569529:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233435.569535:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000568f2a96 x1846780895141952/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233435.569541:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233435.569543:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233435.569544:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233435.569547:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 5996 00000100:00080000:1.0:1761233435.569548:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233435.569551:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233435.569554:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233435.569559:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000038209f5 x1846780895142016/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233435.569565:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233435.569567:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233435.569568:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233435.569570:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5996 00000100:00080000:1.0:1761233435.569572:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233435.569574:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233435.569578:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233435.569579:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895141824:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233435.569586:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000c736bb2 x1846780895142080/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233435.569593:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233435.569596:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233435.569597:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233435.569600:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5996 00000100:00080000:1.0:1761233435.569602:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233435.569604:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233435.569607:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233435.569616:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a95774fb x1846780895142144/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233435.569621:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233435.569623:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233435.569624:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233435.569626:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 5996 00000100:00080000:1.0:1761233435.569627:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233435.569630:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233435.569632:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233435.569639:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cb5be845 x1846780895142208/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233435.569652:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233435.569673:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233435/real 1761233435] req@000000007ba3340f x1846780895141824/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233460 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233435.569686:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007ba3340f x1846780895141824/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233460 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233435.569694:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233435.569697:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233435.569700:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895141824:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233435.569705:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895141888:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233435.569720:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233435.569725:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233435/real 1761233435] req@00000000c652d0ad x1846780895141888/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233460 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233435.569732:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c652d0ad x1846780895141888/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233460 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233435.569737:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233435.569738:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233435.569740:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895141888:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233435.569744:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895141952:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233435.569756:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233435.569761:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233435/real 1761233435] req@00000000568f2a96 x1846780895141952/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233460 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233435.569768:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000568f2a96 x1846780895141952/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233460 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233435.569772:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233435.569774:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233435.569775:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895141952:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233435.569779:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895142016:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233435.569789:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233435.569793:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233435/real 1761233435] req@00000000038209f5 x1846780895142016/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233460 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233435.569799:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000038209f5 x1846780895142016/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233460 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233435.569804:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233435.569806:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233435.569808:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895142016:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233435.569834:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895142080:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233435.569846:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233435.569850:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233435/real 1761233435] req@000000000c736bb2 x1846780895142080/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233460 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233435.569857:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000000c736bb2 x1846780895142080/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233460 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233435.569862:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233435.569864:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233435.569865:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895142080:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233435.569869:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895142144:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233435.569879:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233435.569883:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233435/real 1761233435] req@00000000a95774fb x1846780895142144/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233460 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233435.569889:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a95774fb x1846780895142144/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233460 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233435.569893:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233435.569895:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233435.569897:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895142144:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233435.569900:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895142208:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233435.569910:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233435.569913:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233435/real 1761233435] req@00000000cb5be845 x1846780895142208/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233460 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233435.569920:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cb5be845 x1846780895142208/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233460 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233435.569924:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233435.569926:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233435.569927:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895142208:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233436.337522:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233437.361498:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233438.385474:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233439.409440:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233440.433500:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233440.689434:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233440.689461:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233440.689465:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233440.689476:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6001 00000100:00080000:0.0:1761233440.689479:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233440.689486:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233440.689521:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895142272/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233440.689551:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233440.689554:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233440.689555:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233440.689559:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6001 00000100:00080000:0.0:1761233440.689561:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233440.689565:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233440.689574:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233440.689584:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895142336/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233440.689592:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233440.689595:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233440.689596:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233440.689599:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6001 00000100:00080000:0.0:1761233440.689600:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233440.689602:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233440.689607:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233440.689613:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895142272:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761233440.689617:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895142400/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233440.689629:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233440.689631:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233440.689631:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233440.689634:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6001 00000100:00080000:0.0:1761233440.689636:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233440.689638:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233440.689641:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233440.689650:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895142464/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233440.689656:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233440.689659:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233440.689660:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233440.689662:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6001 00000100:00080000:0.0:1761233440.689664:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233440.689666:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233440.689669:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233440.689676:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895142528/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233440.689682:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233440.689685:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233440.689687:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233440.689689:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6001 00000100:00080000:0.0:1761233440.689690:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233440.689693:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233440.689695:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761233440.689703:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233440.689705:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895142592/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233440.689718:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233440.689722:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233440.689723:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233440.689742:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6001 00000100:00080000:0.0:1761233440.689744:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233440.689749:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761233440.689755:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233440/real 1761233440] req@00000000c6e06344 x1846780895142272/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233465 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:0.0:1761233440.689756:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233440.689773:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895142656/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233440.689790:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895142272/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233465 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233440.689807:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233440.689812:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233440.689818:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895142272:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233440.689829:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895142336:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233440.689862:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233440.689875:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233440/real 1761233440] req@000000005593c994 x1846780895142336/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233465 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233440.689939:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895142336/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233465 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233440.689950:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233440.689954:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233440.689958:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895142336:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233440.689993:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895142400:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233440.690025:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233440.690037:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233440/real 1761233440] req@000000001f9ec359 x1846780895142400/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233465 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233440.690053:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895142400/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233465 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233440.690063:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233440.690067:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233440.690071:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895142400:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233440.690080:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895142464:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233440.690106:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233440.690117:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233440/real 1761233440] req@00000000efa29eb9 x1846780895142464/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233465 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233440.690132:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895142464/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233465 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233440.690142:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233440.690145:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233440.690149:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895142464:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233440.690156:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895142528:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233440.690184:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233440.690196:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233440/real 1761233440] req@000000006d61dd06 x1846780895142528/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233465 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233440.690245:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895142528/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233465 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233440.690270:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233440.690273:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233440.690277:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895142528:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233440.690284:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895142592:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233440.690310:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233440.690320:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233440/real 1761233440] req@00000000ef655201 x1846780895142592/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233465 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233440.690332:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895142592/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233465 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233440.690348:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233440.690351:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233440.690355:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895142592:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233440.690362:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895142656:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233440.690385:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233440.690394:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233440/real 1761233440] req@000000004130b6bd x1846780895142656/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233465 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233440.690407:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895142656/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233465 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233440.690417:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233440.690420:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233440.690424:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895142656:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233441.457428:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233442.481379:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233443.505400:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233444.529344:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233445.553520:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233445.809496:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233445.809509:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233445.809514:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233445.809527:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6006 00000100:00080000:1.0:1761233445.809531:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233445.809541:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233445.809588:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000880f16b4 x1846780895142720/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233445.809630:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233445.809636:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233445.809638:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233445.809642:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6006 00000100:00080000:1.0:1761233445.809645:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233445.809649:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233445.809661:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233445.809677:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000053e52331 x1846780895142784/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233445.809688:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233445.809693:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233445.809695:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233445.809700:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6006 00000100:00080000:1.0:1761233445.809702:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233445.809705:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233445.809712:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895142720:192.168.201.157@tcp:250:kworker/u8:2.0 00000008:00100000:1.0:1761233445.809713:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233445.809730:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090ac513e x1846780895142848/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233445.809743:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233445.809747:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233445.809749:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233445.809753:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6006 00000100:00080000:1.0:1761233445.809756:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233445.809759:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233445.809766:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233445.809779:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d66930c2 x1846780895142912/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233445.809788:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233445.809793:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233445.809796:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233445.809823:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6006 00000100:00080000:1.0:1761233445.809826:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1761233445.809834:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:1.0:1761233445.809835:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233445.809845:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233445.809861:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ffe86d04 x1846780895142976/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233445.809874:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233445/real 1761233445] req@00000000880f16b4 x1846780895142720/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233470 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233445.809876:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233445.809888:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233445.809892:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233445.809906:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6006 00000100:00080000:1.0:1761233445.809909:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233445.809915:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233445.809920:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233445.809923:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000880f16b4 x1846780895142720/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233470 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233445.809937:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df5b2270 x1846780895143040/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233445.809945:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233445.809951:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233445.809954:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:1.0:1761233445.809958:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233445.809963:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233445.809966:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895142720:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:1.0:1761233445.809973:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6006 00000100:00080000:1.0:1761233445.809980:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233445.809989:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895142784:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:1.0:1761233445.809990:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233445.810002:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233445.810021:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df0b94a8 x1846780895143104/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233445.810040:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233445.810055:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233445/real 1761233445] req@0000000053e52331 x1846780895142784/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233470 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233445.810072:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000053e52331 x1846780895142784/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233470 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233445.810081:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233445.810085:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233445.810088:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895142784:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233445.810121:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895142848:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233445.810142:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233445.810150:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233445/real 1761233445] req@0000000090ac513e x1846780895142848/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233470 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233445.810162:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090ac513e x1846780895142848/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233470 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233445.810170:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233445.810172:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233445.810175:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895142848:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233445.810182:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895142912:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233445.810270:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233445.810282:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233445/real 1761233445] req@00000000d66930c2 x1846780895142912/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233470 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233445.810295:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d66930c2 x1846780895142912/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233470 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233445.810304:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233445.810307:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233445.810310:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895142912:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233445.810317:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895142976:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233445.810345:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233445.810353:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233445/real 1761233445] req@00000000ffe86d04 x1846780895142976/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233470 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233445.810365:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ffe86d04 x1846780895142976/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233470 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233445.810373:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233445.810376:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233445.810379:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895142976:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233445.810385:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895143040:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233445.810402:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233445.810409:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233445/real 1761233445] req@00000000df5b2270 x1846780895143040/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233470 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233445.810420:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df5b2270 x1846780895143040/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233470 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233445.810429:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233445.810431:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233445.810434:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895143040:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233445.810439:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895143104:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233445.810456:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233445.810463:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233445/real 1761233445] req@00000000df0b94a8 x1846780895143104/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233470 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233445.810474:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df0b94a8 x1846780895143104/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233470 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233445.810482:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233445.810485:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233445.810488:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895143104:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233446.577418:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233447.601471:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233448.625439:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233449.649444:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233450.673514:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233450.929290:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233450.929302:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233450.929306:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233450.929318:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6011 00000100:00080000:1.0:1761233450.929322:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233450.929328:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233450.929361:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004361f727 x1846780895143168/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233450.929389:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233450.929392:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233450.929393:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233450.929395:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6011 00000100:00080000:1.0:1761233450.929397:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233450.929399:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233450.929408:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233450.929416:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006a5e9865 x1846780895143232/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233450.929421:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233450.929424:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233450.929425:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233450.929428:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6011 00000100:00080000:1.0:1761233450.929430:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233450.929433:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233450.929437:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233450.929445:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac65ecb7 x1846780895143296/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233450.929453:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233450.929457:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233450.929459:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233450.929464:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6011 00000100:00080000:1.0:1761233450.929466:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233450.929470:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233450.929476:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233450.929488:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004361f727 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895143168:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761233450.929491:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df0b94a8 x1846780895143360/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233450.929504:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233450.929508:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233450.929510:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233450.929515:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6011 00000100:00080000:1.0:1761233450.929517:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233450.929521:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233450.929527:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233450.929539:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df5b2270 x1846780895143424/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233450.929546:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233450.929550:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233450.929551:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1761233450.929556:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:1.0:1761233450.929557:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6011 00000100:00080000:1.0:1761233450.929561:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233450.929566:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233450.929571:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233450.929581:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233450/real 1761233450] req@000000004361f727 x1846780895143168/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233475 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233450.929585:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ffe86d04 x1846780895143488/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233450.929606:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1761233450.929610:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004361f727 x1846780895143168/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233475 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233450.929612:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233450.929615:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233450.929623:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6011 00000100:00080000:0.0:1761233450.929624:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233450.929627:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233450.929629:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:1.0:1761233450.929634:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:0.0:1761233450.929636:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004361f727 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895143168:192.168.201.157@tcp:250:kworker/u8:2.0 00000008:00100000:1.0:1761233450.929643:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233450.929645:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a5e9865 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895143232:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233450.929662:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d66930c2 x1846780895143552/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:0.0:1761233450.929669:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233450.929678:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233450/real 1761233450] req@000000006a5e9865 x1846780895143232/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233475 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233450.929692:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006a5e9865 x1846780895143232/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233475 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233450.929698:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233450.929699:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233450.929702:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a5e9865 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895143232:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761233450.929705:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac65ecb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895143296:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233450.929717:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233450.929743:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233450/real 1761233450] req@00000000ac65ecb7 x1846780895143296/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233475 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233450.929750:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac65ecb7 x1846780895143296/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233475 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233450.929755:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233450.929757:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233450.929759:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac65ecb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895143296:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233450.929781:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895143360:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233450.929794:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233450.929799:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233450/real 1761233450] req@00000000df0b94a8 x1846780895143360/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233475 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233450.929806:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df0b94a8 x1846780895143360/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233475 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233450.929810:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233450.929812:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233450.929814:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895143360:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233450.929818:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895143424:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761233450.929829:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233450.929833:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233450/real 1761233450] req@00000000df5b2270 x1846780895143424/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233475 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233450.929839:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df5b2270 x1846780895143424/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233475 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233450.929844:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233450.929845:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233450.929847:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895143424:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761233450.929850:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895143488:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233450.929860:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233450.929864:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233450/real 1761233450] req@00000000ffe86d04 x1846780895143488/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233475 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233450.929870:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ffe86d04 x1846780895143488/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233475 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233450.929875:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233450.929876:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233450.929878:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895143488:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233450.929881:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895143552:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233450.929894:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233450.929898:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233450/real 1761233450] req@00000000d66930c2 x1846780895143552/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233475 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233450.929904:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d66930c2 x1846780895143552/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233475 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233450.929909:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233450.929910:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233450.929912:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895143552:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233451.697507:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233452.721496:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233453.745469:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233454.769453:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233455.793505:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761233456.049432:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233456.049444:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761233456.049448:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233456.049460:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6016 00000100:00080000:3.0:1761233456.049465:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233456.049472:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761233456.049518:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b40830d x1846780895143616/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233456.049567:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233456.049572:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233456.049574:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233456.049579:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6016 00000100:00080000:3.0:1761233456.049581:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233456.049586:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233456.049597:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233456.049613:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000829a5595 x1846780895143680/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233456.049624:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233456.049628:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233456.049630:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233456.049634:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6016 00000100:00080000:3.0:1761233456.049637:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1761233456.049674:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895143616:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:3.0:1761233456.049697:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233456.049704:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233456.049720:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000089b8c4f1 x1846780895143744/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233456.049732:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233456.049736:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233456.049737:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233456.049740:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6016 00000100:00080000:3.0:1761233456.049742:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233456.049745:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233456.049750:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233456.049759:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000070fa3bd6 x1846780895143808/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233456.049768:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233456.049773:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233456.049774:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1761233456.049779:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761233456.049782:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6016 00000100:00080000:3.0:1761233456.049787:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233456.049792:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233456.049798:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233456.049810:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233456/real 1761233456] req@000000003b40830d x1846780895143616/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233481 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233456.049815:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090702bc4 x1846780895143872/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233456.049835:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1761233456.049837:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895143616/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233481 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233456.049843:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233456.049847:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233456.049852:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233456.049855:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6016 00000100:00080000:0.0:1761233456.049857:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:3.0:1761233456.049860:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1761233456.049864:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895143616:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:3.0:1761233456.049869:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:0.0:1761233456.049873:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895143680:192.168.201.157@tcp:38:kworker/u8:3.0 00000008:00100000:3.0:1761233456.049878:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233456.049912:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6fdd7f9 x1846780895143936/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:0.0:1761233456.049916:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233456.049925:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233456/real 1761233456] req@00000000829a5595 x1846780895143680/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233481 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233456.049929:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233456.049941:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1761233456.049944:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895143680/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233481 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233456.049945:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233456.049955:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6016 00000100:00080000:3.0:1761233456.049961:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233456.049967:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233456.049969:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:0.0:1761233456.049974:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000008:00100000:3.0:1761233456.049977:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233456.049982:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895143680:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233456.049992:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008e58c4b8 x1846780895144000/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233456.050023:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895143744:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233456.050053:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233456.050088:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233456/real 1761233456] req@0000000089b8c4f1 x1846780895143744/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233481 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233456.050100:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895143744/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233481 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233456.050106:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233456.050107:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233456.050109:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895143744:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233456.050113:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895143808:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233456.050129:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233456.050134:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233456/real 1761233456] req@0000000070fa3bd6 x1846780895143808/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233481 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233456.050141:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895143808/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233481 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233456.050145:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233456.050147:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233456.050148:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895143808:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233456.050152:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895143872:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761233456.050164:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233456.050168:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233456/real 1761233456] req@0000000090702bc4 x1846780895143872/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233481 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233456.050175:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895143872/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233481 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233456.050180:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233456.050181:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233456.050183:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895143872:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233456.050186:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895143936:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233456.050196:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233456.050251:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233456/real 1761233456] req@00000000b6fdd7f9 x1846780895143936/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233481 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233456.050259:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895143936/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233481 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233456.050264:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233456.050265:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233456.050268:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895143936:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233456.050271:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895144000:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233456.050287:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233456.050297:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233456/real 1761233456] req@000000008e58c4b8 x1846780895144000/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233481 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233456.050306:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895144000/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233481 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233456.050312:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233456.050313:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233456.050319:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895144000:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233456.817472:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233457.841401:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233458.865469:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233459.889407:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233460.913505:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761233461.169355:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233461.169366:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761233461.169371:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233461.169384:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6022 00000100:00080000:3.0:1761233461.169389:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233461.169397:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761233461.169438:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000080131ea7 x1846780895144064/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233461.169466:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233461.169470:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233461.169472:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233461.169475:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6022 00000100:00080000:3.0:1761233461.169477:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233461.169480:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233461.169489:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233461.169499:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007adf99ab x1846780895144128/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233461.169506:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233461.169509:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233461.169511:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233461.169514:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6022 00000100:00080000:3.0:1761233461.169516:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233461.169520:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233461.169524:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233461.169531:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e99e5a12 x1846780895144192/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233461.169537:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233461.169540:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233461.169542:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233461.169544:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6022 00000100:00080000:3.0:1761233461.169547:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233461.169550:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233461.169554:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233461.169585:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008e58c4b8 x1846780895144256/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233461.169593:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233461.169597:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233461.169599:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233461.169603:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6022 00000100:00080000:3.0:1761233461.169605:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233461.169614:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:0.0:1761233461.169615:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895144064:192.168.201.157@tcp:250:kworker/u8:3.0 00000008:00100000:3.0:1761233461.169620:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233461.169631:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6fdd7f9 x1846780895144320/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233461.169640:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233461.169643:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233461.169645:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233461.169649:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6022 00000100:00080000:3.0:1761233461.169651:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233461.169655:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233461.169658:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233461.169670:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090702bc4 x1846780895144384/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233461.169676:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233461.169678:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233461.169679:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233461.169683:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6022 00000100:00080000:3.0:1761233461.169685:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233461.169688:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233461.169692:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1761233461.169698:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233461.169704:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000070fa3bd6 x1846780895144448/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233461.169728:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233461/real 1761233461] req@0000000080131ea7 x1846780895144064/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233486 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233461.169748:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895144064/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233486 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233461.169757:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233461.169760:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233461.169764:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895144064:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761233461.169771:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895144128:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761233461.169790:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233461.169797:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233461/real 1761233461] req@000000007adf99ab x1846780895144128/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233486 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233461.169806:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895144128/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233486 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233461.169813:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233461.169815:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233461.169818:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895144128:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233461.169823:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895144192:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233461.169838:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233461.169843:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233461/real 1761233461] req@00000000e99e5a12 x1846780895144192/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233486 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233461.169852:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895144192/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233486 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233461.169859:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233461.169861:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233461.169863:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895144192:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233461.169867:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895144256:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233461.169880:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233461.169886:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233461/real 1761233461] req@000000008e58c4b8 x1846780895144256/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233486 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233461.169894:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895144256/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233486 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233461.169901:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233461.169910:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233461.169913:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895144256:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233461.169943:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895144320:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761233461.169962:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233461.169969:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233461/real 1761233461] req@00000000b6fdd7f9 x1846780895144320/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233486 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233461.169978:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895144320/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233486 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233461.169985:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233461.169987:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233461.169990:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895144320:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233461.169994:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895144384:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233461.170007:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233461.170012:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233461/real 1761233461] req@0000000090702bc4 x1846780895144384/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233486 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233461.170021:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895144384/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233486 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233461.170027:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233461.170029:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233461.170031:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895144384:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233461.170035:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895144448:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233461.170048:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233461.170053:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233461/real 1761233461] req@0000000070fa3bd6 x1846780895144448/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233486 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233461.170061:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895144448/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233486 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233461.170068:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233461.170070:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233461.170072:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895144448:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233461.937352:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233462.961365:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233463.985422:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233465.009472:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233466.033421:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233466.289337:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233466.289363:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233466.289368:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233466.289383:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6027 00000100:00080000:2.0:1761233466.289388:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233466.289396:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233466.289453:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895144512/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233466.289499:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233466.289504:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233466.289506:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233466.289537:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6027 00000100:00080000:2.0:1761233466.289540:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233466.289545:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233466.289558:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233466.289569:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895144512:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761233466.289574:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895144576/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233466.289588:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233466.289592:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233466.289594:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233466.289599:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6027 00000100:00080000:2.0:1761233466.289602:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233466.289606:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233466.289611:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233466.289624:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895144640/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:0.0:1761233466.289634:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233466.289637:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233466.289642:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233466.289645:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233466.289651:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6027 00000100:00080000:2.0:1761233466.289653:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1761233466.289657:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233466/real 1761233466] req@000000004f29ca26 x1846780895144512/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233491 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233466.289659:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233466.289668:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233466.289676:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895144512/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233491 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233466.289684:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895144704/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233466.289686:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233466.289691:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233466.289698:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895144512:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761233466.289703:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233466.289708:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233466.289711:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233466.289716:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6027 00000100:00080000:2.0:1761233466.289719:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1761233466.289735:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895144576:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:2.0:1761233466.289737:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233466.289748:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1761233466.289761:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233466.289765:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895144768/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233466.289770:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233466/real 1761233466] req@0000000033765045 x1846780895144576/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233491 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233466.289795:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1761233466.289797:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895144576/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233491 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233466.289804:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233466.289808:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233466.289819:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233466.289825:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761233466.289828:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6027 00000100:00100000:0.0:1761233466.289830:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895144576:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:2.0:1761233466.289833:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1761233466.289838:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895144640:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761233466.289841:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233466.289850:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1761233466.289857:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233466.289867:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233466/real 1761233466] req@000000009d315fa5 x1846780895144640/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233491 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233466.289867:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895144832/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233466.289902:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1761233466.289904:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895144640/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233491 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233466.289910:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233466.289914:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233466.289915:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233466.289920:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761233466.289923:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6027 00000100:00100000:0.0:1761233466.289925:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895144640:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761233466.289928:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1761233466.289932:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895144704:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761233466.289936:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233466.289944:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1761233466.289950:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233466.289960:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895144896/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233466.289961:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233466/real 1761233466] req@0000000040d5f361 x1846780895144704/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233491 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233466.289993:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895144704/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233491 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233466.290011:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233466.290014:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233466.290017:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895144704:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233466.290031:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895144768:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761233466.290043:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233466.290053:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233466/real 1761233466] req@000000007dcf84ae x1846780895144768/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233491 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233466.290061:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895144768/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233491 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233466.290066:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233466.290068:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233466.290070:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895144768:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233466.290073:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895144832:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233466.290083:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233466.290088:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233466/real 1761233466] req@000000007091f547 x1846780895144832/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233491 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233466.290094:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895144832/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233491 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233466.290099:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233466.290100:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233466.290101:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895144832:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233466.290105:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895144896:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233466.290113:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233466.290117:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233466/real 1761233466] req@0000000064a2cea0 x1846780895144896/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233491 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233466.290123:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895144896/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233491 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233466.290127:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233466.290129:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233466.290131:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895144896:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233467.057490:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233468.081436:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233469.105416:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233470.129595:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233471.153392:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761233471.409351:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233471.409360:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761233471.409363:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233471.409373:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6032 00000100:00080000:3.0:1761233471.409376:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233471.409381:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761233471.409439:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000089b8c4f1 x1846780895144960/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233471.409461:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233471.409463:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233471.409465:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233471.409467:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6032 00000100:00080000:3.0:1761233471.409469:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233471.409472:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233471.409480:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233471.409488:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000829a5595 x1846780895145024/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233471.409494:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233471.409496:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233471.409498:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233471.409501:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6032 00000100:00080000:3.0:1761233471.409503:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233471.409505:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233471.409507:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233471.409512:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895144960:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233471.409515:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b40830d x1846780895145088/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233471.409524:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233471.409533:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233471.409535:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233471.409537:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6032 00000100:00080000:3.0:1761233471.409539:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233471.409541:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233471.409544:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233471.409552:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a0ba65f x1846780895145152/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233471.409558:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233471.409560:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233471.409561:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233471.409563:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6032 00000100:00080000:3.0:1761233471.409564:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233471.409570:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233471.409573:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233471.409581:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e55ffa x1846780895145216/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:0.0:1761233471.409586:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761233471.409595:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233471.409599:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233471.409601:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233471.409606:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6032 00000100:00080000:3.0:1761233471.409608:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1761233471.409611:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233471/real 1761233471] req@0000000089b8c4f1 x1846780895144960/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233496 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233471.409616:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233471.409622:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233471.409631:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895144960/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233496 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233471.409638:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007aee79ba x1846780895145280/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233471.409640:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233471.409645:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233471.409651:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895144960:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:3.0:1761233471.409657:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1761233471.409660:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895145024:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:3.0:1761233471.409662:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233471.409665:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233471.409670:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6032 00000100:00080000:3.0:1761233471.409671:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233471.409674:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233471.409678:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1761233471.409681:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233471.409688:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233471/real 1761233471] req@00000000829a5595 x1846780895145024/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233496 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233471.409693:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009706a782 x1846780895145344/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233471.409705:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895145024/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233496 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233471.409711:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233471.409713:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233471.409715:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895145024:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233471.409736:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895145088:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233471.409750:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233471.409754:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233471/real 1761233471] req@000000003b40830d x1846780895145088/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233496 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233471.409760:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895145088/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233496 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233471.409765:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233471.409766:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233471.409768:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895145088:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233471.409771:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895145152:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233471.409779:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233471.409783:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233471/real 1761233471] req@000000002a0ba65f x1846780895145152/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233496 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233471.409789:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895145152/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233496 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233471.409793:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233471.409794:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233471.409796:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895145152:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233471.409799:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895145216:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761233471.409808:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233471.409811:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233471/real 1761233471] req@0000000057e55ffa x1846780895145216/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233496 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233471.409817:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895145216/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233496 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233471.409821:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233471.409822:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233471.409824:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895145216:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233471.409827:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895145280:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233471.409836:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233471.409840:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233471/real 1761233471] req@000000007aee79ba x1846780895145280/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233496 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233471.409845:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895145280/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233496 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233471.409849:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233471.409850:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233471.409851:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895145280:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233471.409854:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895145344:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233471.409863:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233471.409866:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233471/real 1761233471] req@000000009706a782 x1846780895145344/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233496 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233471.409872:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895145344/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233496 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233471.409876:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233471.409878:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233471.409879:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895145344:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233472.177438:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233473.201453:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233474.225491:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233475.249534:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233476.273433:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761233476.529365:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233476.529375:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761233476.529379:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233476.529389:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6037 00000100:00080000:3.0:1761233476.529393:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233476.529399:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761233476.529430:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c9993cd4 x1846780895145408/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233476.529454:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233476.529457:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233476.529459:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233476.529461:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6037 00000100:00080000:3.0:1761233476.529463:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233476.529465:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233476.529472:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233476.529481:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002625ec0b x1846780895145472/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233476.529487:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233476.529490:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233476.529491:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233476.529494:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6037 00000100:00080000:3.0:1761233476.529496:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233476.529497:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233476.529501:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233476.529504:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895145408:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233476.529510:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000bbf6af7b x1846780895145536/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233476.529516:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233476.529518:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233476.529519:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233476.529522:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6037 00000100:00080000:3.0:1761233476.529524:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233476.529526:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233476.529529:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233476.529537:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009706a782 x1846780895145600/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233476.529544:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233476.529546:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233476.529547:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233476.529550:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6037 00000100:00080000:3.0:1761233476.529551:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233476.529553:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233476.529556:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233476.529561:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007aee79ba x1846780895145664/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233476.529566:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233476.529568:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233476.529569:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233476.529572:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6037 00000100:00080000:3.0:1761233476.529573:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233476.529575:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233476.529578:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233476.529585:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e55ffa x1846780895145728/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233476.529590:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1761233476.529593:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895145472:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:3.0:1761233476.529594:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233476.529595:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233476.529601:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6037 00000100:00080000:3.0:1761233476.529603:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233476.529606:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233476.529610:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233476.529618:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a0ba65f x1846780895145792/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233476.529644:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895145536:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233476.529665:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895145600:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233476.529679:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895145664:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233476.529693:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895145728:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233476.529708:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895145792:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233477.297384:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233478.321450:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233479.345450:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1761233479.601570:0:2210:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:1.0:1761233479.601630:0:2210:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1761233479.601649:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233479.601702:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233479.601714:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233479.601723:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233479.601730:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233479.601736:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233479.601743:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:0.0:1761233479.601791:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233476/real 1761233479] req@00000000bbf6af7b x1846780895145536/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233501 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233479.601824:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895145536/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233501 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233479.601838:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233479.601843:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233479.601848:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895145536:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233479.601857:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233476/real 1761233479] req@000000009706a782 x1846780895145600/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233501 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233479.601870:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895145600/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233501 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233479.601878:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233479.601882:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233479.601886:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895145600:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233479.601892:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233476/real 1761233479] req@000000007aee79ba x1846780895145664/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233501 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233479.601905:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895145664/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233501 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233479.601913:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233479.601916:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233479.601920:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895145664:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233479.601926:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233476/real 1761233479] req@0000000057e55ffa x1846780895145728/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233501 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233479.601939:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895145728/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233501 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233479.601947:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233479.601951:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233479.601954:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895145728:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233479.601961:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233476/real 1761233479] req@000000002a0ba65f x1846780895145792/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233501 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233479.601973:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895145792/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233501 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233479.601982:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233479.601985:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233479.601989:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895145792:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233479.601996:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233476/real 1761233479] req@00000000c9993cd4 x1846780895145408/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233501 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233479.602009:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895145408/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233501 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233479.602019:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233479.602022:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233479.602026:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895145408:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761233479.602033:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233476/real 1761233479] req@000000002625ec0b x1846780895145472/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233501 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233479.602046:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895145472/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233501 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233479.602057:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233479.602059:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233479.602062:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895145472:192.168.201.157@tcp:38:kworker/u8:3.0 00010000:00010000:3.0:1761233480.369458:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233481.393381:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761233481.649378:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233481.649395:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761233481.649401:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233481.649419:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6042 00000100:00080000:3.0:1761233481.649424:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233481.649436:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761233481.649495:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b40830d x1846780895145856/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233481.649523:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233481.649528:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233481.649530:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233481.649534:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6042 00000100:00080000:3.0:1761233481.649537:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233481.649541:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233481.649551:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233481.649563:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895145856:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233481.649565:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000829a5595 x1846780895145920/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233481.649586:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233481.649590:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233481.649591:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233481.649594:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6042 00000100:00080000:3.0:1761233481.649596:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233481.649599:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233481.649602:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233481.649612:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000089b8c4f1 x1846780895145984/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233481.649619:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233481.649622:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233481.649624:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233481.649627:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6042 00000100:00080000:3.0:1761233481.649629:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233481.649632:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233481.649635:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233481.649643:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000070fa3bd6 x1846780895146048/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233481.649650:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233481.649652:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233481.649653:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233481.649656:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6042 00000100:00080000:3.0:1761233481.649658:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233481.649665:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:0.0:1761233481.649666:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:3.0:1761233481.649671:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233481.649684:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090702bc4 x1846780895146112/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233481.649692:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233481.649695:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233481.649697:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1761233481.649701:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233481/real 1761233481] req@000000003b40830d x1846780895145856/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233506 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233481.649705:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6042 00000100:00080000:3.0:1761233481.649710:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233481.649717:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233481.649722:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233481.649735:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6fdd7f9 x1846780895146176/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233481.649741:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1761233481.649742:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895145856/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233506 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233481.649758:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233481.649761:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233481.649762:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:3.0:1761233481.649765:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233481.649771:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6042 00000100:00100000:0.0:1761233481.649775:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895145856:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:3.0:1761233481.649776:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233481.649783:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233481.649788:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233481.649798:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008e58c4b8 x1846780895146240/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233481.649814:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895145920:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761233481.649839:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233481.649847:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233481/real 1761233481] req@00000000829a5595 x1846780895145920/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233506 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233481.649858:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895145920/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233506 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233481.649868:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233481.649872:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233481.649877:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895145920:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233481.649884:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895145984:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233481.649913:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233481.649924:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233481/real 1761233481] req@0000000089b8c4f1 x1846780895145984/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233506 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233481.649937:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895145984/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233506 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233481.649946:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233481.649949:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233481.649953:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895145984:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233481.649961:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895146048:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233481.649990:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233481.650001:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233481/real 1761233481] req@0000000070fa3bd6 x1846780895146048/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233506 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233481.650017:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895146048/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233506 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233481.650026:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233481.650029:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233481.650033:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895146048:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233481.650040:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895146112:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761233481.650066:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233481.650076:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233481/real 1761233481] req@0000000090702bc4 x1846780895146112/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233506 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233481.650090:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895146112/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233506 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233481.650099:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233481.650101:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233481.650105:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895146112:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233481.650112:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895146176:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233481.650172:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233481.650185:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233481/real 1761233481] req@00000000b6fdd7f9 x1846780895146176/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233506 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233481.650198:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895146176/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233506 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233481.650295:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233481.650299:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233481.650304:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895146176:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233481.650338:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895146240:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233481.650388:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233481.650402:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233481/real 1761233481] req@000000008e58c4b8 x1846780895146240/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233506 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233481.650417:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895146240/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233506 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233481.650440:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233481.650443:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233481.650448:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895146240:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233482.417454:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233483.441479:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233484.465458:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233485.489383:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233486.513445:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233486.769333:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233486.769342:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233486.769345:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233486.769353:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6047 00000100:00080000:0.0:1761233486.769356:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233486.769361:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233486.769402:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895146304/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233486.769423:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233486.769426:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233486.769428:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233486.769430:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6047 00000100:00080000:0.0:1761233486.769432:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233486.769434:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233486.769441:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233486.769448:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895146368/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233486.769454:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233486.769457:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233486.769458:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233486.769461:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6047 00000100:00080000:0.0:1761233486.769462:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233486.769464:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233486.769466:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233486.769472:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895146304:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233486.769474:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895146432/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233486.769485:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233486.769487:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233486.769487:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233486.769489:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6047 00000100:00080000:0.0:1761233486.769490:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233486.769492:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233486.769494:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233486.769503:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895146496/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233486.769508:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233486.769511:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233486.769512:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233486.769514:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6047 00000100:00080000:0.0:1761233486.769515:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233486.769517:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233486.769520:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233486.769526:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895146560/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233486.769530:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233486.769531:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233486.769532:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233486.769535:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6047 00000100:00080000:0.0:1761233486.769536:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233486.769538:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233486.769541:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233486.769549:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895146624/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233486.769550:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233486.769556:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233486.769559:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233486.769560:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233486.769563:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6047 00000100:00080000:0.0:1761233486.769564:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233486.769567:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233486.769570:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233486.769578:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233486/real 1761233486] req@000000009d1c7f73 x1846780895146304/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233511 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233486.769579:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895146688/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233486.769612:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895146304/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233511 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233486.769621:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233486.769624:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233486.769628:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895146304:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233486.769636:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895146368:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233486.769659:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233486.769677:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233486/real 1761233486] req@00000000d67bb589 x1846780895146368/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233511 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233486.769688:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895146368/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233511 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233486.769696:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233486.769699:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233486.769702:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895146368:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233486.769726:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895146432:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233486.769749:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233486.769758:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233486/real 1761233486] req@00000000a4a2c44a x1846780895146432/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233511 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233486.769769:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895146432/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233511 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233486.769776:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233486.769779:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233486.769781:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895146432:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233486.769798:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895146496:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233486.769817:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233486.769825:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233486/real 1761233486] req@000000004a74da19 x1846780895146496/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233511 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233486.769836:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895146496/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233511 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233486.769843:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233486.769846:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233486.769849:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895146496:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233486.769854:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895146560:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233486.769873:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233486.769881:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233486/real 1761233486] req@000000002d1b8ef7 x1846780895146560/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233511 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233486.769892:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895146560/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233511 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233486.769899:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233486.769902:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233486.769904:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895146560:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233486.769909:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895146624:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233486.769928:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233486.769936:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233486/real 1761233486] req@000000007bd1ec64 x1846780895146624/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233511 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233486.769946:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895146624/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233511 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233486.769952:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233486.769954:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233486.769957:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895146624:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233486.769962:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895146688:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233486.769980:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233486.769988:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233486/real 1761233486] req@000000001db96463 x1846780895146688/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233511 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233486.769997:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895146688/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233511 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233486.770004:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233486.770006:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233486.770009:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895146688:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233487.537359:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233488.561380:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233489.585440:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233490.609446:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233491.633360:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233491.889291:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233491.889302:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233491.889306:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233491.889316:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6052 00000100:00080000:0.0:1761233491.889320:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233491.889328:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233491.889364:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895146752/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233491.889387:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233491.889390:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233491.889392:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233491.889395:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6052 00000100:00080000:0.0:1761233491.889396:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233491.889399:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233491.889406:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233491.889415:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895146816/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233491.889420:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233491.889422:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233491.889423:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233491.889426:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6052 00000100:00080000:0.0:1761233491.889427:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233491.889429:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233491.889432:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233491.889438:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895146880/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233491.889443:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233491.889444:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233491.889445:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233491.889447:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6052 00000100:00080000:0.0:1761233491.889448:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233491.889450:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233491.889453:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233491.889459:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895146944/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233491.889466:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233491.889469:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233491.889470:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233491.889474:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6052 00000100:00080000:0.0:1761233491.889476:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233491.889481:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233491.889481:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895146752:192.168.201.157@tcp:250:kworker/u8:2.0 00000008:00100000:0.0:1761233491.889489:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233491.889503:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895147008/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233491.889520:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233491.889524:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233491.889526:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233491.889530:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6052 00000100:00080000:0.0:1761233491.889533:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233491.889537:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233491.889542:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233491.889542:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233491.889566:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895147072/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233491.889574:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233491/real 1761233491] req@000000005d83493f x1846780895146752/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233516 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233491.889575:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233491.889582:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233491.889585:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233491.889591:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6052 00000100:00080000:0.0:1761233491.889592:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233491.889596:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005d83493f x1846780895146752/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233516 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233491.889597:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233491.889603:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761233491.889610:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233491.889614:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233491.889616:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895147136/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233491.889621:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895146752:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233491.889632:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895146816:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233491.889648:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233491.889654:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233491/real 1761233491] req@00000000f3ed2588 x1846780895146816/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233516 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233491.889663:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f3ed2588 x1846780895146816/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233516 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233491.889668:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233491.889670:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233491.889672:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895146816:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233491.889677:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895146880:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233491.889690:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233491.889697:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233491/real 1761233491] req@000000001db96463 x1846780895146880/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233516 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233491.889709:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895146880/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233516 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233491.889714:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233491.889715:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233491.889716:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895146880:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233491.889720:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895146944:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233491.889729:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233491.889733:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233491/real 1761233491] req@000000007bd1ec64 x1846780895146944/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233516 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233491.889739:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895146944/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233516 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233491.889743:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233491.889744:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233491.889745:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895146944:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233491.889767:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895147008:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233491.889776:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233491.889781:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233491/real 1761233491] req@000000002d1b8ef7 x1846780895147008/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233516 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233491.889786:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895147008/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233516 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233491.889790:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233491.889792:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233491.889793:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895147008:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233491.889797:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895147072:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233491.889805:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233491.889809:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233491/real 1761233491] req@000000004a74da19 x1846780895147072/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233516 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233491.889814:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895147072/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233516 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233491.889819:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233491.889820:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233491.889822:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895147072:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233491.889825:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895147136:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233491.889833:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233491.889836:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233491/real 1761233491] req@00000000a4a2c44a x1846780895147136/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233516 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233491.889842:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895147136/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233516 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233491.889846:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233491.889847:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233491.889849:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895147136:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233492.657385:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233493.681406:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233494.705386:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233495.729429:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233496.753388:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233497.009307:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233497.009316:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233497.009319:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233497.009330:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6057 00000100:00080000:0.0:1761233497.009333:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233497.009341:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233497.009374:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895147200/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233497.009397:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233497.009401:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233497.009403:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233497.009406:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6057 00000100:00080000:0.0:1761233497.009408:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233497.009412:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233497.009422:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233497.009433:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895147264/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233497.009442:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233497.009445:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233497.009446:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233497.009450:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6057 00000100:00080000:0.0:1761233497.009452:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233497.009455:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233497.009459:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233497.009459:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895147200:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233497.009467:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895147328/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233497.009475:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233497.009477:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233497.009478:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233497.009482:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6057 00000100:00080000:0.0:1761233497.009484:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233497.009486:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233497.009490:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233497.009497:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895147392/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233497.009503:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233497.009506:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233497.009507:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233497.009509:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6057 00000100:00080000:0.0:1761233497.009511:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233497.009514:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233497.009517:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233497.009524:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895147456/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233497.009524:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233497.009540:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233497.009543:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233497.009544:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233497.009547:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6057 00000100:00080000:0.0:1761233497.009549:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233497.009551:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233497.009555:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233497.009556:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233497/real 1761233497] req@00000000d67bb589 x1846780895147200/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233522 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233497.009562:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895147520/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233497.009572:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895147200/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233522 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233497.009574:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233497.009577:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233497.009579:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233497.009581:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6057 00000100:00080000:3.0:1761233497.009582:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233497.009583:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233497.009585:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233497.009586:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233497.009588:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895147200:192.168.201.157@tcp:250:kworker/u8:2.0 00000008:00100000:0.0:1761233497.009601:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233497.009606:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895147264:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761233497.009608:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895147584/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233497.009627:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233497.009635:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233497/real 1761233497] req@000000009d1c7f73 x1846780895147264/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233522 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233497.009645:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895147264/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233522 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233497.009653:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233497.009655:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233497.009658:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895147264:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233497.009682:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895147328:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233497.009699:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233497.009705:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233497/real 1761233497] req@000000004130b6bd x1846780895147328/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233522 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233497.009713:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895147328/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233522 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233497.009717:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233497.009719:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233497.009721:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895147328:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233497.009724:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895147392:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233497.009733:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233497.009737:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233497/real 1761233497] req@00000000ef655201 x1846780895147392/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233522 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233497.009742:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895147392/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233522 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233497.009747:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233497.009748:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233497.009749:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895147392:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233497.009753:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895147456:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233497.009762:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233497.009766:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233497/real 1761233497] req@000000006d61dd06 x1846780895147456/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233522 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233497.009771:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895147456/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233522 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233497.009776:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233497.009777:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233497.009779:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895147456:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233497.009782:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895147520:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233497.009791:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233497.009795:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233497/real 1761233497] req@00000000efa29eb9 x1846780895147520/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233522 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233497.009800:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895147520/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233522 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233497.009805:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233497.009806:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233497.009808:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895147520:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233497.009810:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895147584:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233497.009819:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233497.009822:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233497/real 1761233497] req@000000001f9ec359 x1846780895147584/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233522 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233497.009827:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895147584/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233522 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233497.009832:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233497.009833:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233497.009835:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895147584:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233497.777511:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233498.801414:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233499.825395:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233500.849421:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233501.873351:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233502.129315:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233502.129324:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233502.129327:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233502.129338:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6063 00000100:00080000:0.0:1761233502.129341:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233502.129347:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233502.129379:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895147648/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233502.129402:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233502.129404:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233502.129405:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233502.129408:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6063 00000100:00080000:0.0:1761233502.129409:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233502.129412:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233502.129421:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233502.129428:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895147712/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233502.129434:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233502.129437:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233502.129438:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233502.129440:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6063 00000100:00080000:0.0:1761233502.129441:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233502.129443:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233502.129446:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233502.129451:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895147776/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233502.129456:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233502.129458:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233502.129459:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233502.129463:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6063 00000100:00080000:0.0:1761233502.129465:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233502.129465:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895147648:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233502.129469:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233502.129474:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233502.129480:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895147840/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233502.129485:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233502.129487:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233502.129488:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233502.129491:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6063 00000100:00080000:0.0:1761233502.129493:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233502.129496:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233502.129499:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233502.129507:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895147904/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233502.129512:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233502.129513:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233502.129514:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233502.129517:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6063 00000100:00080000:0.0:1761233502.129518:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233502.129520:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233502.129523:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233502.129528:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895147968/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233502.129533:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233502.129535:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233502.129535:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233502.129537:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6063 00000100:00080000:0.0:1761233502.129538:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233502.129541:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233502.129544:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233502.129548:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233502.129552:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895148032/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233502.129575:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233502/real 1761233502] req@000000005593c994 x1846780895147648/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233527 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233502.129593:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895147648/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233527 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233502.129603:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233502.129607:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233502.129611:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895147648:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233502.129618:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895147712:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233502.129641:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233502.129659:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233502/real 1761233502] req@00000000c6e06344 x1846780895147712/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233527 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233502.129669:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895147712/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233527 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233502.129677:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233502.129679:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233502.129682:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895147712:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233502.129710:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895147776:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233502.129731:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233502.129740:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233502/real 1761233502] req@000000004862b808 x1846780895147776/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233527 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233502.129753:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895147776/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233527 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233502.129760:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233502.129762:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233502.129765:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895147776:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233502.129771:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895147840:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233502.129790:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233502.129798:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233502/real 1761233502] req@000000006f2efc8b x1846780895147840/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233527 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233502.129808:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895147840/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233527 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233502.129816:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233502.129818:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233502.129821:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895147840:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233502.129826:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895147904:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233502.129845:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233502.129853:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233502/real 1761233502] req@00000000eca9ddc2 x1846780895147904/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233527 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233502.129863:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895147904/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233527 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233502.129871:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233502.129875:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233502.129878:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895147904:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233502.129884:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895147968:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233502.129904:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233502.129912:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233502/real 1761233502] req@0000000034d49f59 x1846780895147968/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233527 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233502.129943:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895147968/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233527 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233502.129949:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233502.129951:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233502.129954:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895147968:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233502.129959:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895148032:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233502.129979:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233502.129987:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233502/real 1761233502] req@000000005beaae31 x1846780895148032/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233527 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233502.129998:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895148032/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233527 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233502.130006:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233502.130008:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233502.130011:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895148032:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233502.897371:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233503.921415:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233504.945384:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233505.969439:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233506.993517:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761233507.249401:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233507.249413:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761233507.249418:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233507.249431:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6068 00000100:00080000:3.0:1761233507.249435:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233507.249443:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761233507.249500:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e99e5a12 x1846780895148096/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233507.249541:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233507.249546:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233507.249549:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233507.249554:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6068 00000100:00080000:3.0:1761233507.249558:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233507.249564:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233507.249579:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233507.249582:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895148096:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233507.249598:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007adf99ab x1846780895148160/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233507.249611:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233507.249615:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233507.249618:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233507.249622:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6068 00000100:00080000:3.0:1761233507.249625:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233507.249629:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233507.249637:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233507.249648:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000080131ea7 x1846780895148224/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233507.249657:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233507.249661:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233507.249663:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233507.249667:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6068 00000800:00000100:0.0:1761233507.249671:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761233507.249671:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233507.249679:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233507.249686:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233507.249703:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233507/real 1761233507] req@00000000e99e5a12 x1846780895148096/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233532 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233507.249704:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008e58c4b8 x1846780895148288/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233507.249730:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233507.249735:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1761233507.249736:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895148096/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233532 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233507.249738:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233507.249748:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6068 00000100:00080000:0.0:1761233507.249753:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233507.249753:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233507.249759:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:3.0:1761233507.249762:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:0.0:1761233507.249767:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895148096:192.168.201.157@tcp:250:kworker/u8:3.0 00000008:00100000:3.0:1761233507.249769:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233507.249784:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6fdd7f9 x1846780895148352/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233507.249792:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233507.249795:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233507.249797:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1761233507.249800:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895148160:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:3.0:1761233507.249804:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6068 00000100:00080000:3.0:1761233507.249808:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233507.249812:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233507.249817:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1761233507.249828:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233507.249849:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090702bc4 x1846780895148416/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233507.249855:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233507/real 1761233507] req@000000007adf99ab x1846780895148160/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233532 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233507.249886:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895148160/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233532 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233507.249886:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233507.249897:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233507.249902:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233507.249903:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233507.249909:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00080000:3.0:1761233507.249912:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6068 00000100:00100000:0.0:1761233507.249915:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895148160:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:3.0:1761233507.249919:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1761233507.249924:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895148224:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1761233507.249929:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233507.249937:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1761233507.249949:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233507.249956:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000070fa3bd6 x1846780895148480/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233507.249963:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233507/real 1761233507] req@0000000080131ea7 x1846780895148224/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233532 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233507.249982:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895148224/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233532 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233507.249992:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233507.249994:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233507.249999:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895148224:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233507.250007:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895148288:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233507.250031:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233507.250040:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233507/real 1761233507] req@000000008e58c4b8 x1846780895148288/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233532 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233507.250054:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895148288/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233532 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233507.250062:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233507.250065:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233507.250068:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895148288:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233507.250075:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895148352:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761233507.250099:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233507.250106:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233507/real 1761233507] req@00000000b6fdd7f9 x1846780895148352/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233532 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233507.250115:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895148352/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233532 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233507.250121:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233507.250123:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233507.250125:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895148352:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233507.250141:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895148416:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233507.250155:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233507.250160:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233507/real 1761233507] req@0000000090702bc4 x1846780895148416/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233532 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233507.250168:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895148416/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233532 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233507.250173:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233507.250174:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233507.250176:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895148416:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233507.250180:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895148480:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233507.250190:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233507.250195:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233507/real 1761233507] req@0000000070fa3bd6 x1846780895148480/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233532 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233507.250225:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895148480/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233532 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233507.250231:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233507.250233:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233507.250235:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895148480:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233508.017545:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233509.041386:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233510.065461:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233511.089440:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233512.113439:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761233512.369290:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233512.369298:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761233512.369302:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233512.369312:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6073 00000100:00080000:3.0:1761233512.369316:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233512.369321:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761233512.369352:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000089b8c4f1 x1846780895148544/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233512.369382:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233512.369385:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233512.369386:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233512.369388:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6073 00000100:00080000:3.0:1761233512.369390:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233512.369392:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233512.369398:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233512.369405:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000829a5595 x1846780895148608/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233512.369410:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233512.369413:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233512.369414:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233512.369416:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6073 00000100:00080000:3.0:1761233512.369417:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233512.369419:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233512.369422:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233512.369428:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b40830d x1846780895148672/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233512.369433:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233512.369435:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233512.369436:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233512.369438:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6073 00000100:00080000:3.0:1761233512.369440:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1761233512.369443:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895148544:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:3.0:1761233512.369445:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233512.369451:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233512.369458:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002625ec0b x1846780895148736/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233512.369463:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233512.369465:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233512.369467:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233512.369469:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6073 00000100:00080000:3.0:1761233512.369470:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233512.369472:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233512.369475:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233512.369484:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c9993cd4 x1846780895148800/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233512.369489:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233512.369491:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233512.369492:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233512.369494:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6073 00000100:00080000:3.0:1761233512.369496:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233512.369498:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233512.369499:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233512.369506:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a0ba65f x1846780895148864/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233512.369512:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1761233512.369513:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761233512.369515:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233512.369518:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233512.369522:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6073 00000100:00080000:3.0:1761233512.369524:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233512.369526:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233512.369528:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233512.369534:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e55ffa x1846780895148928/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233512.369538:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233512/real 1761233512] req@0000000089b8c4f1 x1846780895148544/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233537 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233512.369558:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895148544/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233537 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233512.369568:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233512.369571:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233512.369575:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895148544:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761233512.369582:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895148608:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761233512.369606:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233512.369615:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233512/real 1761233512] req@00000000829a5595 x1846780895148608/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233537 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233512.369627:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895148608/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233537 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233512.369634:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233512.369637:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233512.369640:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895148608:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233512.369663:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895148672:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233512.369684:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233512.369693:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233512/real 1761233512] req@000000003b40830d x1846780895148672/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233537 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233512.369714:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895148672/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233537 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233512.369723:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233512.369724:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233512.369727:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895148672:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233512.369731:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895148736:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233512.369745:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233512.369750:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233512/real 1761233512] req@000000002625ec0b x1846780895148736/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233537 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233512.369783:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895148736/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233537 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233512.369787:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233512.369789:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233512.369791:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895148736:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233512.369805:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895148800:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761233512.369818:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233512.369823:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233512/real 1761233512] req@00000000c9993cd4 x1846780895148800/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233537 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233512.369830:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895148800/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233537 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233512.369834:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233512.369835:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233512.369837:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895148800:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233512.369841:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895148864:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233512.369850:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233512.369854:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233512/real 1761233512] req@000000002a0ba65f x1846780895148864/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233537 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233512.369861:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895148864/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233537 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233512.369867:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233512.369870:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233512.369873:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895148864:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233512.369878:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895148928:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233512.369896:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233512.369903:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233512/real 1761233512] req@0000000057e55ffa x1846780895148928/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233537 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233512.369914:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895148928/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233537 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233512.369922:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233512.369924:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233512.369926:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895148928:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233513.137481:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233514.161543:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233515.185466:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233516.209398:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233517.233439:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761233517.489355:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233517.489364:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761233517.489367:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233517.489377:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6078 00000100:00080000:3.0:1761233517.489380:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233517.489386:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761233517.489417:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007aee79ba x1846780895148992/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233517.489440:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233517.489443:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233517.489444:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233517.489447:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6078 00000100:00080000:3.0:1761233517.489448:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233517.489450:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233517.489457:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233517.489464:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009706a782 x1846780895149056/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233517.489470:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233517.489472:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233517.489473:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233517.489475:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6078 00000100:00080000:3.0:1761233517.489476:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233517.489478:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233517.489485:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233517.489490:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000bbf6af7b x1846780895149120/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233517.489494:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233517.489496:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233517.489497:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233517.489499:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6078 00000100:00080000:3.0:1761233517.489501:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233517.489502:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233517.489505:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233517.489512:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e55ffa x1846780895149184/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233517.489516:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233517.489518:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233517.489519:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233517.489522:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6078 00000100:00080000:3.0:1761233517.489524:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233517.489526:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233517.489529:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233517.489535:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a0ba65f x1846780895149248/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233517.489540:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1761233517.489544:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895148992:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:3.0:1761233517.489544:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233517.489545:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233517.489551:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6078 00000100:00080000:3.0:1761233517.489554:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233517.489572:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233517.489576:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233517.489586:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c9993cd4 x1846780895149312/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233517.489593:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233517.489595:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233517.489597:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233517.489600:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6078 00000100:00080000:3.0:1761233517.489602:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233517.489605:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233517.489608:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233517.489617:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002625ec0b x1846780895149376/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:0.0:1761233517.489649:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233517.489700:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233517/real 1761233517] req@000000007aee79ba x1846780895148992/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233542 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233517.489730:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895148992/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233542 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233517.489737:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233517.489739:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233517.489742:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895148992:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233517.489747:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895149056:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761233517.489769:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233517.489777:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233517/real 1761233517] req@000000009706a782 x1846780895149056/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233542 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233517.489788:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895149056/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233542 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233517.489795:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233517.489797:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233517.489800:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895149056:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761233517.489806:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895149120:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233517.489827:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233517.489835:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233517/real 1761233517] req@00000000bbf6af7b x1846780895149120/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233542 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233517.489842:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895149120/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233542 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233517.489847:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233517.489849:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233517.489850:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895149120:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233517.489854:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895149184:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233517.489867:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233517.489871:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233517/real 1761233517] req@0000000057e55ffa x1846780895149184/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233542 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233517.489877:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895149184/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233542 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233517.489881:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233517.489882:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233517.489884:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895149184:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233517.489905:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895149248:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761233517.489915:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233517.489919:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233517/real 1761233517] req@000000002a0ba65f x1846780895149248/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233542 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233517.489943:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895149248/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233542 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233517.489949:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233517.489951:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233517.489954:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895149248:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761233517.489958:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895149312:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233517.489974:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233517.489980:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233517/real 1761233517] req@00000000c9993cd4 x1846780895149312/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233542 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233517.489996:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895149312/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233542 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233517.490002:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233517.490003:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233517.490005:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895149312:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233517.490009:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895149376:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233517.490021:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233517.490025:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233517/real 1761233517] req@000000002625ec0b x1846780895149376/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233542 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233517.490033:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895149376/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233542 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233517.490039:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233517.490041:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233517.490043:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895149376:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233518.257434:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233519.281461:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233520.305447:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233521.329430:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233522.353390:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761233522.609320:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233522.609330:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761233522.609334:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233522.609344:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6083 00000100:00080000:3.0:1761233522.609347:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233522.609352:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761233522.609386:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b40830d x1846780895149440/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233522.609410:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233522.609413:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233522.609414:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233522.609418:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6083 00000100:00080000:3.0:1761233522.609419:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233522.609423:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233522.609432:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233522.609440:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000829a5595 x1846780895149504/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233522.609448:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233522.609451:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895149440:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:3.0:1761233522.609452:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233522.609454:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233522.609461:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6083 00000100:00080000:3.0:1761233522.609462:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233522.609464:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233522.609468:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233522.609478:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000089b8c4f1 x1846780895149568/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233522.609484:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233522.609487:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233522.609488:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233522.609491:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6083 00000100:00080000:3.0:1761233522.609492:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233522.609494:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233522.609496:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233522.609503:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000070fa3bd6 x1846780895149632/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233522.609508:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233522.609510:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233522.609511:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233522.609513:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6083 00000100:00080000:3.0:1761233522.609514:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1761233522.609518:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761233522.609519:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233522.609524:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233522.609531:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090702bc4 x1846780895149696/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233522.609537:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233522.609539:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1761233522.609542:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233522/real 1761233522] req@000000003b40830d x1846780895149440/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233547 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233522.609542:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233522.609551:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6083 00000100:00080000:3.0:1761233522.609555:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233522.609560:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233522.609565:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233522.609574:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6fdd7f9 x1846780895149760/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233522.609579:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895149440/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233547 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233522.609585:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233522.609590:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233522.609606:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233522.609607:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233522.609612:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:3.0:1761233522.609612:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6083 00000100:00080000:3.0:1761233522.609615:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761233522.609619:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895149440:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:3.0:1761233522.609622:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233522.609628:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233522.609638:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008e58c4b8 x1846780895149824/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233522.609650:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895149504:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233522.609669:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233522.609675:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233522/real 1761233522] req@00000000829a5595 x1846780895149504/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233547 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233522.609682:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895149504/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233547 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233522.609687:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233522.609689:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233522.609691:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895149504:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233522.609694:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895149568:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233522.609706:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233522.609710:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233522/real 1761233522] req@0000000089b8c4f1 x1846780895149568/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233547 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233522.609716:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895149568/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233547 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233522.609721:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233522.609723:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233522.609725:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895149568:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233522.609728:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895149632:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233522.609738:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233522.609742:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233522/real 1761233522] req@0000000070fa3bd6 x1846780895149632/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233547 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233522.609748:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895149632/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233547 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233522.609753:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233522.609754:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233522.609756:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895149632:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233522.609759:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895149696:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233522.609769:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233522.609773:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233522/real 1761233522] req@0000000090702bc4 x1846780895149696/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233547 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233522.609780:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895149696/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233547 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233522.609785:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233522.609786:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233522.609788:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895149696:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233522.609791:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895149760:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233522.609803:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233522.609806:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233522/real 1761233522] req@00000000b6fdd7f9 x1846780895149760/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233547 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233522.609812:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895149760/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233547 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233522.609817:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233522.609819:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233522.609820:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895149760:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233522.609823:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895149824:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233522.609833:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233522.609837:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233522/real 1761233522] req@000000008e58c4b8 x1846780895149824/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233547 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233522.609843:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895149824/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233547 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233522.609855:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233522.609857:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233522.609859:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895149824:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233523.377419:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233524.401366:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233525.425432:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233526.449357:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233527.473419:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761233527.729306:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233527.729315:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761233527.729318:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233527.729329:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6088 00000100:00080000:3.0:1761233527.729332:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233527.729337:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761233527.729370:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000080131ea7 x1846780895149888/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233527.729395:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233527.729397:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233527.729399:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233527.729401:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6088 00000100:00080000:3.0:1761233527.729402:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233527.729405:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233527.729412:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233527.729420:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007adf99ab x1846780895149952/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233527.729426:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233527.729429:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895149888:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:3.0:1761233527.729430:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233527.729432:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233527.729439:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6088 00000100:00080000:3.0:1761233527.729441:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233527.729442:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233527.729445:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233527.729452:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e99e5a12 x1846780895150016/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233527.729457:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233527.729460:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233527.729461:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233527.729463:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6088 00000100:00080000:3.0:1761233527.729465:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233527.729467:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233527.729470:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233527.729478:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008e58c4b8 x1846780895150080/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233527.729483:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233527.729485:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233527.729486:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233527.729488:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6088 00000100:00080000:3.0:1761233527.729489:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233527.729492:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233527.729494:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233527.729499:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6fdd7f9 x1846780895150144/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233527.729503:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233527.729505:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233527.729506:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1761233527.729524:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761233527.729525:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6088 00000100:00080000:3.0:1761233527.729529:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233527.729531:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233527.729534:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233527.729542:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090702bc4 x1846780895150208/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233527.729548:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233527.729550:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233527.729552:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761233527.729555:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233527/real 1761233527] req@0000000080131ea7 x1846780895149888/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233552 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233527.729558:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6088 00000100:00080000:3.0:1761233527.729561:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233527.729565:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233527.729570:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233527.729578:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000070fa3bd6 x1846780895150272/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233527.729588:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895149888/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233552 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233527.729601:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233527.729605:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233527.729609:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895149888:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233527.729633:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895149952:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233527.729660:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233527.729669:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233527/real 1761233527] req@000000007adf99ab x1846780895149952/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233552 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233527.729681:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895149952/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233552 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233527.729689:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233527.729692:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233527.729697:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895149952:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233527.729703:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895150016:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233527.729725:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233527.729733:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233527/real 1761233527] req@00000000e99e5a12 x1846780895150016/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233552 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233527.729744:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895150016/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233552 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233527.729752:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233527.729755:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233527.729758:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895150016:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233527.729764:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895150080:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233527.729786:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233527.729794:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233527/real 1761233527] req@000000008e58c4b8 x1846780895150080/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233552 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233527.729805:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895150080/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233552 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233527.729812:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233527.729814:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233527.729817:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895150080:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233527.729823:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895150144:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233527.729843:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233527.729851:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233527/real 1761233527] req@00000000b6fdd7f9 x1846780895150144/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233552 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233527.729861:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895150144/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233552 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233527.729869:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233527.729871:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233527.729874:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895150144:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233527.729880:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895150208:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233527.729903:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233527.729911:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233527/real 1761233527] req@0000000090702bc4 x1846780895150208/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233552 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233527.729922:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895150208/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233552 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233527.729930:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233527.729932:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233527.729935:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895150208:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233527.729940:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895150272:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233527.729960:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233527.729969:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233527/real 1761233527] req@0000000070fa3bd6 x1846780895150272/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233552 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233527.729980:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895150272/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233552 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233527.729988:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233527.729990:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233527.729993:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895150272:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233528.497414:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233529.521445:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233530.545441:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233531.569371:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233532.593445:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761233532.849334:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233532.849346:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761233532.849350:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233532.849361:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6093 00000100:00080000:3.0:1761233532.849364:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233532.849371:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761233532.849402:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000089b8c4f1 x1846780895150336/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233532.849423:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233532.849426:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233532.849427:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233532.849429:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6093 00000100:00080000:3.0:1761233532.849442:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233532.849445:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233532.849453:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233532.849462:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000829a5595 x1846780895150400/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233532.849469:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895150336:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:3.0:1761233532.849470:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233532.849476:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233532.849478:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233532.849482:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6093 00000100:00080000:3.0:1761233532.849483:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233532.849485:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233532.849488:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233532.849497:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b40830d x1846780895150464/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233532.849502:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233532.849504:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233532.849505:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233532.849508:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6093 00000100:00080000:3.0:1761233532.849509:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233532.849511:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233532.849514:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233532.849520:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002625ec0b x1846780895150528/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233532.849524:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233532.849526:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233532.849527:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233532.849529:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6093 00000100:00080000:3.0:1761233532.849531:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1761233532.849536:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761233532.849536:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233532.849540:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233532.849547:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c9993cd4 x1846780895150592/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233532.849553:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233532.849555:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233532.849556:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761233532.849559:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233532/real 1761233532] req@0000000089b8c4f1 x1846780895150336/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233557 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233532.849561:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6093 00000100:00080000:3.0:1761233532.849564:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233532.849569:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233532.849573:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233532.849581:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a0ba65f x1846780895150656/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233532.849584:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895150336/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233557 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233532.849597:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233532.849601:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233532.849602:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233532.849603:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233532.849608:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:3.0:1761233532.849610:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6093 00000100:00080000:3.0:1761233532.849613:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761233532.849614:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895150336:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:3.0:1761233532.849617:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233532.849623:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233532.849630:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e55ffa x1846780895150720/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233532.849640:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895150400:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233532.849657:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233532.849662:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233532/real 1761233532] req@00000000829a5595 x1846780895150400/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233557 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233532.849669:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895150400/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233557 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233532.849673:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233532.849674:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233532.849676:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895150400:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233532.849680:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895150464:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233532.849690:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233532.849694:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233532/real 1761233532] req@000000003b40830d x1846780895150464/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233557 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233532.849700:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895150464/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233557 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233532.849708:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233532.849709:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233532.849711:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895150464:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233532.849714:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895150528:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233532.849723:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233532.849727:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233532/real 1761233532] req@000000002625ec0b x1846780895150528/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233557 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233532.849732:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895150528/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233557 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233532.849736:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233532.849737:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233532.849738:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895150528:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233532.849742:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895150592:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233532.849751:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233532.849754:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233532/real 1761233532] req@00000000c9993cd4 x1846780895150592/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233557 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233532.849761:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895150592/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233557 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233532.849765:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233532.849766:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233532.849767:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895150592:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233532.849771:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895150656:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233532.849780:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233532.849784:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233532/real 1761233532] req@000000002a0ba65f x1846780895150656/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233557 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233532.849789:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895150656/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233557 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233532.849793:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233532.849795:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233532.849796:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895150656:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233532.849799:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895150720:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233532.849808:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233532.849811:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233532/real 1761233532] req@0000000057e55ffa x1846780895150720/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233557 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233532.849817:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895150720/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233557 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233532.849821:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233532.849822:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233532.849824:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895150720:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233533.617430:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233534.641393:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233535.665386:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233536.689448:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233537.713412:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761233537.969315:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233537.969325:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761233537.969328:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233537.969338:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6098 00000100:00080000:3.0:1761233537.969342:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233537.969348:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761233537.969408:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000bbf6af7b x1846780895150784/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233537.969430:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233537.969433:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233537.969434:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233537.969437:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6098 00000100:00080000:3.0:1761233537.969438:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233537.969440:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233537.969448:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233537.969456:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009706a782 x1846780895150848/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233537.969463:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233537.969465:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233537.969466:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233537.969471:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6098 00000100:00100000:1.0:1761233537.969473:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895150784:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:3.0:1761233537.969474:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233537.969477:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233537.969482:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233537.969488:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007aee79ba x1846780895150912/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233537.969493:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233537.969495:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233537.969496:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233537.969498:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6098 00000100:00080000:3.0:1761233537.969500:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233537.969503:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233537.969505:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233537.969513:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e55ffa x1846780895150976/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233537.969518:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233537.969520:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233537.969521:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233537.969523:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6098 00000100:00080000:3.0:1761233537.969525:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233537.969527:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233537.969530:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233537.969535:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a0ba65f x1846780895151040/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233537.969540:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233537.969541:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233537.969542:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233537.969544:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6098 00000100:00080000:3.0:1761233537.969546:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1761233537.969551:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761233537.969552:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233537.969557:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233537.969563:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c9993cd4 x1846780895151104/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233537.969569:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233537.969571:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233537.969572:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233537.969574:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6098 00000100:00080000:3.0:1761233537.969576:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233537.969577:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233537.969580:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233537.969586:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002625ec0b x1846780895151168/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233537.969591:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233537/real 1761233537] req@00000000bbf6af7b x1846780895150784/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233562 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233537.969615:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895150784/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233562 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233537.969626:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233537.969629:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233537.969634:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895150784:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761233537.969659:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895150848:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233537.969686:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233537.969695:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233537/real 1761233537] req@000000009706a782 x1846780895150848/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233562 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233537.969708:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895150848/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233562 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233537.969717:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233537.969719:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233537.969723:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895150848:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233537.969729:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895150912:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233537.969751:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233537.969759:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233537/real 1761233537] req@000000007aee79ba x1846780895150912/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233562 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233537.969771:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895150912/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233562 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233537.969785:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233537.969788:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233537.969791:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895150912:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233537.969797:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895150976:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233537.969818:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233537.969826:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233537/real 1761233537] req@0000000057e55ffa x1846780895150976/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233562 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233537.969838:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895150976/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233562 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233537.969845:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233537.969847:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233537.969851:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895150976:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233537.969857:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895151040:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233537.969876:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233537.969884:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233537/real 1761233537] req@000000002a0ba65f x1846780895151040/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233562 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233537.969895:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895151040/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233562 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233537.969902:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233537.969905:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233537.969907:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895151040:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233537.969914:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895151104:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233537.969934:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233537.969942:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233537/real 1761233537] req@00000000c9993cd4 x1846780895151104/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233562 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233537.969953:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895151104/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233562 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233537.969961:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233537.969963:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233537.969966:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895151104:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233537.969971:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895151168:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233537.969990:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233537.969998:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233537/real 1761233537] req@000000002625ec0b x1846780895151168/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233562 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233537.970009:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895151168/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233562 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233537.970017:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233537.970020:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233537.970022:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895151168:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233538.737422:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233539.761367:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233540.785419:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233541.809385:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233542.833436:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233543.089345:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233543.089355:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233543.089359:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233543.089376:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6104 00000100:00080000:1.0:1761233543.089380:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233543.089390:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233543.089427:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090ac513e x1846780895151232/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233543.089454:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233543.089457:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233543.089458:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233543.089461:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6104 00000100:00080000:1.0:1761233543.089462:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233543.089465:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233543.089473:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233543.089482:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000053e52331 x1846780895151296/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233543.089488:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233543.089491:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233543.089492:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233543.089495:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6104 00000100:00080000:1.0:1761233543.089496:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233543.089499:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233543.089502:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233543.089508:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000880f16b4 x1846780895151360/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233543.089514:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895151232:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233543.089517:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233543.089522:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233543.089523:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233543.089526:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6104 00000100:00080000:1.0:1761233543.089527:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233543.089530:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233543.089532:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233543.089543:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cb5be845 x1846780895151424/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233543.089551:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233543.089554:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233543.089555:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233543.089559:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6104 00000100:00080000:1.0:1761233543.089561:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233543.089565:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233543.089570:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233543.089581:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a95774fb x1846780895151488/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233543.089590:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233543.089593:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233543.089594:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233543.089598:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6104 00000100:00080000:1.0:1761233543.089600:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233543.089605:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233543.089609:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233543.089619:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895151296:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233543.089621:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000c736bb2 x1846780895151552/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233543.089635:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233543.089648:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233543.089650:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233543.089655:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6104 00000100:00080000:1.0:1761233543.089658:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233543.089662:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233543.089667:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233543.089678:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895151360:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233543.089681:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000038209f5 x1846780895151616/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233543.089705:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895151424:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233543.089722:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895151488:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233543.089735:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895151552:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233543.089752:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895151616:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233543.857342:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233544.881574:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233545.905506:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:3.0:1761233546.161453:0:2207:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:3.0:1761233546.161492:0:2207:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1761233546.161504:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:3.0:1761233546.161537:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:3.0:1761233546.161544:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:3.0:1761233546.161548:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:3.0:1761233546.161553:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:3.0:1761233546.161559:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:3.0:1761233546.161565:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:0.0:1761233546.161639:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233543/real 1761233546] req@00000000038209f5 x1846780895151616/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233568 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233546.161666:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000038209f5 x1846780895151616/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233568 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233546.161675:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233546.161679:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233546.161683:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895151616:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233546.161690:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233543/real 1761233546] req@00000000880f16b4 x1846780895151360/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233568 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233546.161700:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000880f16b4 x1846780895151360/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233568 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233546.161706:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233546.161709:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233546.161712:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895151360:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233546.161717:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233543/real 1761233546] req@00000000cb5be845 x1846780895151424/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233568 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233546.161728:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cb5be845 x1846780895151424/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233568 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233546.161735:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233546.161738:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233546.161740:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895151424:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233546.161746:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233543/real 1761233546] req@00000000a95774fb x1846780895151488/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233568 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233546.161757:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a95774fb x1846780895151488/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233568 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233546.161764:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233546.161767:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233546.161770:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895151488:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233546.161775:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233543/real 1761233546] req@000000000c736bb2 x1846780895151552/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233568 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233546.161786:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000000c736bb2 x1846780895151552/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233568 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233546.161793:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233546.161795:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233546.161798:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895151552:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233546.161803:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233543/real 1761233546] req@0000000090ac513e x1846780895151232/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233568 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233546.161812:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090ac513e x1846780895151232/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233568 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233546.161824:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233546.161826:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233546.161829:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895151232:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761233546.161834:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233543/real 1761233546] req@0000000053e52331 x1846780895151296/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233568 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233546.161843:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000053e52331 x1846780895151296/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233568 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233546.161850:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233546.161852:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233546.161854:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895151296:192.168.201.157@tcp:38:kworker/u8:3.0 00010000:00010000:1.0:1761233546.929470:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233547.953397:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233548.209382:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233548.209426:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233548.209433:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233548.209448:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6109 00000100:00080000:1.0:1761233548.209454:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233548.209463:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233548.209505:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000568f2a96 x1846780895151680/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233548.209546:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233548.209550:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233548.209553:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233548.209557:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6109 00000100:00080000:1.0:1761233548.209560:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233548.209564:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233548.209574:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233548.209584:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c652d0ad x1846780895151744/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233548.209592:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233548.209595:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233548.209596:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233548.209599:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6109 00000100:00080000:1.0:1761233548.209601:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233548.209604:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233548.209608:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233548.209616:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ba3340f x1846780895151808/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233548.209623:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233548.209626:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233548.209630:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233548.209634:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6109 00000100:00080000:1.0:1761233548.209636:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233548.209639:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233548.209644:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233548.209655:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d74b2e3c x1846780895151872/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233548.209662:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233548.209665:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233548.209666:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233548.209672:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6109 00000100:00080000:1.0:1761233548.209673:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233548.209677:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233548.209680:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233548.209695:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001aedf9b1 x1846780895151936/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233548.209721:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895151680:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233548.209721:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233548.209741:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233548.209744:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233548.209751:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6109 00000100:00080000:1.0:1761233548.209753:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233548.209758:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233548.209764:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233548.209783:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a611ee34 x1846780895152000/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233548.209814:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233548.209819:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233548.209822:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233548.209827:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6109 00000100:00080000:1.0:1761233548.209839:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233548.209847:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233548.209854:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233548.209874:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000328368c8 x1846780895152064/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:0.0:1761233548.209917:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233548.209967:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233548/real 1761233548] req@00000000568f2a96 x1846780895151680/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233573 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233548.210000:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000568f2a96 x1846780895151680/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233573 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233548.210037:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233548.210042:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233548.210049:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895151680:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761233548.210064:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895151744:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761233548.210105:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233548.210122:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233548/real 1761233548] req@00000000c652d0ad x1846780895151744/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233573 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233548.210141:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c652d0ad x1846780895151744/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233573 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233548.210154:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233548.210158:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233548.210164:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895151744:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233548.210174:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895151808:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233548.210361:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233548.210378:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233548/real 1761233548] req@000000007ba3340f x1846780895151808/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233573 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233548.210398:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007ba3340f x1846780895151808/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233573 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233548.210410:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233548.210414:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233548.210420:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895151808:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233548.210430:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895151872:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233548.210465:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233548.210479:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233548/real 1761233548] req@00000000d74b2e3c x1846780895151872/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233573 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233548.210509:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d74b2e3c x1846780895151872/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233573 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233548.210529:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233548.210533:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233548.210539:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895151872:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233548.210598:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895151936:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761233548.210635:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233548.210651:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233548/real 1761233548] req@000000001aedf9b1 x1846780895151936/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233573 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233548.210668:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001aedf9b1 x1846780895151936/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233573 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233548.210678:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233548.210682:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233548.210687:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895151936:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233548.210696:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895152000:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233548.210726:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233548.210738:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233548/real 1761233548] req@00000000a611ee34 x1846780895152000/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233573 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233548.210754:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a611ee34 x1846780895152000/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233573 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233548.210764:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233548.210768:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233548.210772:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895152000:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233548.210783:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895152064:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233548.210815:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233548.210828:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233548/real 1761233548] req@00000000328368c8 x1846780895152064/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233573 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233548.210848:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000328368c8 x1846780895152064/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233573 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233548.210861:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233548.210865:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233548.210872:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895152064:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233548.977427:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233550.001474:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233551.025410:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233552.049474:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233553.073398:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233553.329336:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233553.329346:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233553.329349:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233553.329359:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6114 00000100:00080000:1.0:1761233553.329363:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233553.329370:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233553.329405:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090800dde x1846780895152128/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233553.329430:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233553.329433:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233553.329434:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233553.329438:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6114 00000100:00080000:1.0:1761233553.329439:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233553.329441:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233553.329449:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233553.329458:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e467e107 x1846780895152192/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233553.329465:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233553.329468:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233553.329470:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233553.329473:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6114 00000100:00080000:1.0:1761233553.329475:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233553.329477:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233553.329480:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233553.329487:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cbad49de x1846780895152256/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233553.329494:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1761233553.329497:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090800dde pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895152128:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233553.329498:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233553.329500:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233553.329508:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6114 00000100:00080000:1.0:1761233553.329510:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233553.329512:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233553.329516:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233553.329542:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000328368c8 x1846780895152320/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233553.329547:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233553.329550:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233553.329551:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233553.329553:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6114 00000100:00080000:1.0:1761233553.329555:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233553.329558:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233553.329561:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233553.329570:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a611ee34 x1846780895152384/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233553.329575:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233553.329578:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1761233553.329581:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:1.0:1761233553.329581:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233553.329587:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6114 00000100:00080000:1.0:1761233553.329589:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233553.329592:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233553.329595:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233553.329602:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001aedf9b1 x1846780895152448/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233553.329606:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233553/real 1761233553] req@0000000090800dde x1846780895152128/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233578 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233553.329611:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233553.329617:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233553.329619:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233553.329624:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6114 00000100:00080000:1.0:1761233553.329626:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1761233553.329630:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090800dde x1846780895152128/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233578 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233553.329630:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233553.329635:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1761233553.329644:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1761233553.329645:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d74b2e3c x1846780895152512/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233553.329648:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233553.329654:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090800dde pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895152128:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761233553.329662:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e467e107 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895152192:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761233553.329678:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233553.329683:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233553/real 1761233553] req@00000000e467e107 x1846780895152192/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233578 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233553.329694:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e467e107 x1846780895152192/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233578 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233553.329699:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233553.329700:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233553.329702:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e467e107 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895152192:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233553.329721:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbad49de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895152256:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233553.329733:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233553.329737:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233553/real 1761233553] req@00000000cbad49de x1846780895152256/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233578 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233553.329743:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cbad49de x1846780895152256/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233578 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233553.329748:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233553.329749:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233553.329750:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbad49de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895152256:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233553.329754:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895152320:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233553.329769:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233553.329776:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233553/real 1761233553] req@00000000328368c8 x1846780895152320/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233578 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233553.329787:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000328368c8 x1846780895152320/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233578 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233553.329794:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233553.329797:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233553.329799:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895152320:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233553.329805:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895152384:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761233553.329826:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233553.329834:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233553/real 1761233553] req@00000000a611ee34 x1846780895152384/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233578 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233553.329842:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a611ee34 x1846780895152384/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233578 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233553.329847:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233553.329848:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233553.329850:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895152384:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233553.329853:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895152448:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233553.329866:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233553.329870:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233553/real 1761233553] req@000000001aedf9b1 x1846780895152448/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233578 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233553.329876:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001aedf9b1 x1846780895152448/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233578 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233553.329880:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233553.329881:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233553.329883:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895152448:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233553.329886:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895152512:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233553.329897:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233553.329900:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233553/real 1761233553] req@00000000d74b2e3c x1846780895152512/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233578 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233553.329906:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d74b2e3c x1846780895152512/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233578 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233553.329911:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233553.329912:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233553.329913:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895152512:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233554.097410:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233555.121398:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233556.145421:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233557.169399:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233558.193464:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761233558.449355:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233558.449366:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761233558.449370:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233558.449381:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6119 00000100:00080000:3.0:1761233558.449385:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233558.449392:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761233558.449430:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b40830d x1846780895152576/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233558.449453:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233558.449456:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233558.449457:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233558.449460:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6119 00000100:00080000:3.0:1761233558.449462:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233558.449465:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233558.449474:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233558.449485:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000829a5595 x1846780895152640/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233558.449495:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1761233558.449499:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895152576:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:3.0:1761233558.449500:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233558.449502:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233558.449509:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6119 00000100:00080000:3.0:1761233558.449512:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233558.449514:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233558.449518:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233558.449530:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000089b8c4f1 x1846780895152704/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233558.449536:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233558.449539:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233558.449540:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233558.449544:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6119 00000100:00080000:3.0:1761233558.449545:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233558.449548:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233558.449551:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233558.449559:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000070fa3bd6 x1846780895152768/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233558.449566:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233558.449569:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233558.449570:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233558.449572:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6119 00000100:00080000:3.0:1761233558.449574:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233558.449581:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:0.0:1761233558.449583:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:3.0:1761233558.449587:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233558.449599:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090702bc4 x1846780895152832/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233558.449607:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233558.449610:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233558.449612:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1761233558.449616:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233558/real 1761233558] req@000000003b40830d x1846780895152576/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233583 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233558.449619:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6119 00000100:00080000:3.0:1761233558.449624:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233558.449631:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233558.449638:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233558.449650:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6fdd7f9 x1846780895152896/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233558.449660:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895152576/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233583 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233558.449660:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233558.449668:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233558.449670:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233558.449678:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6119 00000100:00080000:3.0:1761233558.449682:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233558.449687:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233558.449691:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1761233558.449693:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233558.449701:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233558.449704:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008e58c4b8 x1846780895152960/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233558.449712:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895152576:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233558.449748:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895152640:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761233558.449773:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233558.449782:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233558/real 1761233558] req@00000000829a5595 x1846780895152640/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233583 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233558.449792:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895152640/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233583 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233558.449800:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233558.449802:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233558.449805:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895152640:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761233558.449809:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895152704:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233558.449825:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233558.449830:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233558/real 1761233558] req@0000000089b8c4f1 x1846780895152704/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233583 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233558.449843:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895152704/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233583 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233558.449852:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233558.449855:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233558.449859:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895152704:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233558.449867:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895152768:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233558.449895:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233558.449904:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233558/real 1761233558] req@0000000070fa3bd6 x1846780895152768/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233583 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233558.449917:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895152768/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233583 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233558.449926:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233558.449929:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233558.449932:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895152768:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233558.449940:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895152832:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761233558.449963:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233558.449973:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233558/real 1761233558] req@0000000090702bc4 x1846780895152832/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233583 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233558.449985:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895152832/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233583 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233558.449994:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233558.449997:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233558.450000:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895152832:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761233558.450007:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895152896:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233558.450051:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233558.450062:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233558/real 1761233558] req@00000000b6fdd7f9 x1846780895152896/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233583 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233558.450091:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895152896/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233583 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233558.450101:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233558.450104:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233558.450108:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895152896:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233558.450115:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895152960:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233558.450134:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233558.450140:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233558/real 1761233558] req@000000008e58c4b8 x1846780895152960/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233583 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233558.450150:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895152960/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233583 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233558.450155:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233558.450157:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233558.450159:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895152960:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233559.217418:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233560.241436:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233561.265384:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233562.289398:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233563.313384:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233563.569354:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233563.569362:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233563.569365:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233563.569375:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6124 00000100:00080000:0.0:1761233563.569379:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233563.569386:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233563.569425:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000078e7052d x1846780895153024/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233563.569457:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233563.569461:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233563.569463:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233563.569466:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6124 00000100:00080000:0.0:1761233563.569468:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233563.569471:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233563.569482:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233563.569496:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000addc3db8 x1846780895153088/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233563.569501:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895153024:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233563.569514:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233563.569518:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233563.569519:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233563.569523:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6124 00000100:00080000:0.0:1761233563.569525:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233563.569528:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233563.569531:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233563.569542:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895153152/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233563.569547:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233563.569549:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233563.569550:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233563.569552:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6124 00000100:00080000:0.0:1761233563.569553:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233563.569555:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233563.569557:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233563.569565:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895153216/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761233563.569568:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233563.569573:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233563.569576:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233563.569577:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233563.569579:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6124 00000100:00080000:0.0:1761233563.569581:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233563.569584:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233563.569586:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233563.569590:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233563/real 1761233563] req@0000000078e7052d x1846780895153024/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233588 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233563.569594:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895153280/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233563.569613:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233563.569615:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233563.569616:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761233563.569620:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000078e7052d x1846780895153024/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233588 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233563.569621:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6124 00000100:00080000:0.0:1761233563.569625:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233563.569630:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233563.569635:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1761233563.569638:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1761233563.569642:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895153344/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233563.569642:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233563.569648:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895153024:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233563.569654:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233563.569657:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233563.569658:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233563.569661:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6124 00000100:00080000:0.0:1761233563.569662:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233563.569664:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233563.569666:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233563.569672:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895153408/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233563.569674:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895153088:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233563.569697:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233563.569704:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233563/real 1761233563] req@00000000addc3db8 x1846780895153088/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233588 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233563.569712:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000addc3db8 x1846780895153088/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233588 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233563.569716:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233563.569718:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233563.569720:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895153088:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233563.569723:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895153152:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233563.569733:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233563.569737:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233563/real 1761233563] req@000000005beaae31 x1846780895153152/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233588 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233563.569747:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895153152/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233588 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233563.569751:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233563.569752:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233563.569754:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895153152:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233563.569774:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895153216:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233563.569787:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233563.569792:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233563/real 1761233563] req@0000000034d49f59 x1846780895153216/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233588 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233563.569799:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895153216/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233588 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233563.569804:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233563.569806:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233563.569808:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895153216:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233563.569811:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895153280:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233563.569822:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233563.569827:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233563/real 1761233563] req@00000000eca9ddc2 x1846780895153280/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233588 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233563.569833:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895153280/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233588 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233563.569838:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233563.569840:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233563.569842:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895153280:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233563.569847:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895153344:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233563.569857:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233563.569861:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233563/real 1761233563] req@000000006f2efc8b x1846780895153344/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233588 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233563.569868:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895153344/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233588 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233563.569874:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233563.569875:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233563.569877:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895153344:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233563.569893:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895153408:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233563.569904:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233563.569909:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233563/real 1761233563] req@000000004862b808 x1846780895153408/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233588 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233563.569916:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895153408/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233588 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233563.569921:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233563.569923:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233563.569925:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895153408:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233564.337465:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233565.361425:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233566.385365:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233567.409376:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233568.433391:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233568.689366:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233568.689374:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233568.689377:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233568.689387:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6129 00000100:00080000:1.0:1761233568.689390:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233568.689396:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233568.689428:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ba3340f x1846780895153472/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233568.689453:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233568.689456:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233568.689457:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233568.689459:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6129 00000100:00080000:1.0:1761233568.689461:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233568.689463:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233568.689470:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233568.689479:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c652d0ad x1846780895153536/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233568.689485:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233568.689488:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233568.689490:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233568.689493:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6129 00000100:00080000:1.0:1761233568.689494:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233568.689495:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233568.689500:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233568.689510:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000568f2a96 x1846780895153600/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233568.689511:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895153472:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233568.689533:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233568.689535:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233568.689536:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233568.689539:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6129 00000100:00080000:1.0:1761233568.689540:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233568.689541:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233568.689544:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233568.689551:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000053e52331 x1846780895153664/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233568.689556:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233568.689559:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233568.689560:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233568.689563:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6129 00000100:00080000:1.0:1761233568.689564:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233568.689567:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233568.689570:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233568.689576:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090ac513e x1846780895153728/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233568.689584:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233568.689586:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233568.689587:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233568.689589:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6129 00000100:00080000:1.0:1761233568.689591:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233568.689593:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233568.689596:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233568.689602:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000c736bb2 x1846780895153792/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233568.689606:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233568.689608:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233568.689609:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233568.689616:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6129 00000800:00000100:0.0:1761233568.689618:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:1.0:1761233568.689623:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233568.689627:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233568.689630:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233568.689639:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a95774fb x1846780895153856/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233568.689654:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233568/real 1761233568] req@000000007ba3340f x1846780895153472/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233593 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233568.689675:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007ba3340f x1846780895153472/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233593 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233568.689684:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233568.689687:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233568.689690:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895153472:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761233568.689697:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895153536:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761233568.689713:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233568.689718:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233568/real 1761233568] req@00000000c652d0ad x1846780895153536/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233593 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233568.689724:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c652d0ad x1846780895153536/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233593 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233568.689729:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233568.689730:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233568.689732:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895153536:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233568.689757:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895153600:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233568.689774:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233568.689782:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233568/real 1761233568] req@00000000568f2a96 x1846780895153600/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233593 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233568.689793:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000568f2a96 x1846780895153600/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233593 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233568.689801:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233568.689805:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233568.689808:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895153600:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233568.689814:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895153664:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233568.689836:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233568.689845:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233568/real 1761233568] req@0000000053e52331 x1846780895153664/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233593 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233568.689853:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000053e52331 x1846780895153664/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233593 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233568.689858:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233568.689859:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233568.689861:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895153664:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233568.689864:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895153728:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761233568.689896:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233568.689905:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233568/real 1761233568] req@0000000090ac513e x1846780895153728/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233593 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233568.689926:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090ac513e x1846780895153728/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233593 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233568.689934:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233568.689937:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233568.689940:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895153728:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233568.689947:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895153792:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233568.689971:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233568.689980:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233568/real 1761233568] req@000000000c736bb2 x1846780895153792/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233593 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233568.689989:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000000c736bb2 x1846780895153792/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233593 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233568.689994:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233568.689996:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233568.689998:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895153792:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233568.690002:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895153856:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761233568.690016:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233568.690020:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233568/real 1761233568] req@00000000a95774fb x1846780895153856/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233593 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233568.690027:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a95774fb x1846780895153856/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233593 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233568.690031:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233568.690033:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233568.690034:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895153856:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233569.457380:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233570.481410:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233571.505438:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233572.529416:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233573.553470:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761233573.809395:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233573.809408:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761233573.809413:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233573.809426:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6134 00000100:00080000:3.0:1761233573.809431:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233573.809438:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761233573.809485:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e99e5a12 x1846780895153920/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233573.809513:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233573.809516:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233573.809518:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233573.809521:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6134 00000100:00080000:3.0:1761233573.809524:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233573.809527:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233573.809537:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233573.809563:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007adf99ab x1846780895153984/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233573.809572:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233573.809576:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233573.809578:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233573.809583:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6134 00000100:00080000:3.0:1761233573.809586:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233573.809589:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233573.809596:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233573.809601:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895153920:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233573.809612:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000080131ea7 x1846780895154048/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233573.809625:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233573.809628:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233573.809630:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233573.809635:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6134 00000100:00080000:3.0:1761233573.809637:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233573.809640:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233573.809643:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233573.809654:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008e58c4b8 x1846780895154112/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233573.809663:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233573.809665:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233573.809666:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233573.809669:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6134 00000100:00080000:3.0:1761233573.809670:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233573.809672:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233573.809675:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233573.809684:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6fdd7f9 x1846780895154176/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:0.0:1761233573.809695:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761233573.809701:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233573.809704:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233573.809705:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233573.809711:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6134 00000100:00080000:3.0:1761233573.809714:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233573.809718:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233573.809723:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233573.809730:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233573/real 1761233573] req@00000000e99e5a12 x1846780895153920/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233598 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233573.809731:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090702bc4 x1846780895154240/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233573.809819:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895153920/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233598 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233573.809819:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233573.809850:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233573.809852:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233573.809863:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6134 00000100:00080000:3.0:1761233573.809867:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233573.809868:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233573.809871:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:0.0:1761233573.809873:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000008:00100000:3.0:1761233573.809876:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233573.809881:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895153920:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233573.809892:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895153984:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233573.809892:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000070fa3bd6 x1846780895154304/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:0.0:1761233573.809929:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233573.809937:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233573/real 1761233573] req@000000007adf99ab x1846780895153984/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233598 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233573.809945:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895153984/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233598 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233573.809950:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233573.809952:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233573.809954:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895153984:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761233573.809976:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895154048:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233573.809987:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233573.809991:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233573/real 1761233573] req@0000000080131ea7 x1846780895154048/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233598 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233573.809997:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895154048/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233598 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233573.810001:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233573.810003:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233573.810004:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895154048:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233573.810008:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895154112:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233573.810018:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233573.810022:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233573/real 1761233573] req@000000008e58c4b8 x1846780895154112/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233598 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233573.810028:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895154112/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233598 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233573.810032:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233573.810033:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233573.810034:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895154112:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233573.810053:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895154176:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761233573.810067:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233573.810072:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233573/real 1761233573] req@00000000b6fdd7f9 x1846780895154176/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233598 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233573.810079:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895154176/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233598 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233573.810084:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233573.810086:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233573.810088:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895154176:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761233573.810092:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895154240:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233573.810104:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233573.810108:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233573/real 1761233573] req@0000000090702bc4 x1846780895154240/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233598 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233573.810116:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895154240/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233598 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233573.810122:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233573.810123:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233573.810125:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895154240:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233573.810129:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895154304:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233573.810141:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233573.810145:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233573/real 1761233573] req@0000000070fa3bd6 x1846780895154304/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233598 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233573.810152:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895154304/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233598 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233573.810157:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233573.810159:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233573.810161:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895154304:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233574.577409:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233575.601488:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233576.625390:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233577.649516:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233578.673402:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761233578.929326:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233578.929335:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761233578.929339:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233578.929349:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6139 00000100:00080000:3.0:1761233578.929352:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233578.929358:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761233578.929395:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000089b8c4f1 x1846780895154368/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233578.929416:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233578.929419:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233578.929420:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233578.929423:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6139 00000100:00080000:3.0:1761233578.929424:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233578.929426:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233578.929433:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233578.929440:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000829a5595 x1846780895154432/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233578.929445:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233578.929447:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233578.929448:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233578.929450:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6139 00000100:00080000:3.0:1761233578.929452:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233578.929454:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233578.929456:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233578.929462:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b40830d x1846780895154496/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233578.929467:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233578.929468:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233578.929469:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233578.929471:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6139 00000100:00080000:3.0:1761233578.929473:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233578.929475:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233578.929479:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233578.929482:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895154368:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233578.929487:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002625ec0b x1846780895154560/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233578.929494:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233578.929496:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233578.929497:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233578.929499:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6139 00000100:00080000:3.0:1761233578.929500:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233578.929503:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233578.929505:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233578.929511:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c9993cd4 x1846780895154624/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233578.929516:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233578.929518:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233578.929520:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233578.929522:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6139 00000100:00080000:3.0:1761233578.929524:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233578.929526:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233578.929528:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233578.929534:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a0ba65f x1846780895154688/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233578.929539:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233578.929541:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233578.929543:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1761233578.929547:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761233578.929550:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6139 00000100:00080000:3.0:1761233578.929552:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233578.929555:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233578.929558:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233578.929566:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e55ffa x1846780895154752/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761233578.929570:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233578/real 1761233578] req@0000000089b8c4f1 x1846780895154368/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233603 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233578.929586:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895154368/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233603 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233578.929594:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233578.929597:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233578.929600:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895154368:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233578.929605:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895154432:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761233578.929619:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233578.929625:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233578/real 1761233578] req@00000000829a5595 x1846780895154432/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233603 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233578.929636:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895154432/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233603 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233578.929643:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233578.929645:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233578.929648:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895154432:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761233578.929653:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895154496:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233578.929672:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233578.929680:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233578/real 1761233578] req@000000003b40830d x1846780895154496/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233603 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233578.929690:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895154496/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233603 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233578.929696:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233578.929698:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233578.929701:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895154496:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233578.929767:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895154560:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233578.929790:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233578.929798:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233578/real 1761233578] req@000000002625ec0b x1846780895154560/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233603 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233578.929808:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895154560/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233603 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233578.929815:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233578.929816:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233578.929819:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895154560:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233578.929824:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895154624:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:0.0:1761233578.929840:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233578.929848:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233578/real 1761233578] req@00000000c9993cd4 x1846780895154624/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233603 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233578.929858:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895154624/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233603 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233578.929865:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233578.929867:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233578.929870:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895154624:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761233578.929875:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895154688:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233578.929889:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233578.929895:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233578/real 1761233578] req@000000002a0ba65f x1846780895154688/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233603 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233578.929904:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895154688/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233603 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233578.929912:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233578.929914:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233578.929917:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895154688:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:0.0:1761233578.929921:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895154752:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:0.0:1761233578.929934:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233578.929940:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233578/real 1761233578] req@0000000057e55ffa x1846780895154752/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233603 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233578.929949:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895154752/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233603 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233578.929956:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233578.929958:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233578.929961:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895154752:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233579.697499:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233580.721343:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233581.745442:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233582.769368:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233583.793426:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233584.049394:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233584.049405:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233584.049409:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233584.049418:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6144 00000100:00080000:0.0:1761233584.049421:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233584.049426:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233584.049458:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895154816/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233584.049483:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233584.049487:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233584.049490:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233584.049493:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6144 00000100:00080000:0.0:1761233584.049495:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233584.049498:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233584.049507:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233584.049519:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895154880/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233584.049527:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233584.049530:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233584.049532:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233584.049535:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6144 00000100:00080000:0.0:1761233584.049536:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233584.049539:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233584.049543:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233584.049546:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895154816:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233584.049551:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895154944/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233584.049558:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233584.049561:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233584.049562:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233584.049565:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6144 00000100:00080000:0.0:1761233584.049567:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233584.049570:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233584.049573:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233584.049580:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895155008/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233584.049586:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233584.049589:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233584.049590:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233584.049593:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6144 00000100:00080000:0.0:1761233584.049595:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233584.049598:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233584.049601:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233584.049608:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895155072/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761233584.049610:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233584.049615:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233584.049618:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233584.049619:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233584.049622:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6144 00000100:00080000:0.0:1761233584.049624:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233584.049642:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761233584.049646:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233584/real 1761233584] req@00000000c6e06344 x1846780895154816/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233609 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000008:00100000:0.0:1761233584.049647:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233584.049655:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895155136/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233584.049681:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233584.049684:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233584.049686:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761233584.049688:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895154816/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233609 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233584.049690:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6144 00000100:00080000:0.0:1761233584.049692:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233584.049696:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:1.0:1761233584.049699:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1761233584.049700:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1761233584.049702:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233584.049706:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895154816:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233584.049708:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895155200/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233584.049714:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895154880:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233584.049740:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233584.049751:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233584/real 1761233584] req@000000005593c994 x1846780895154880/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233609 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233584.049765:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895154880/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233609 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233584.049772:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233584.049774:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233584.049776:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895154880:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233584.049799:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895154944:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233584.049817:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233584.049823:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233584/real 1761233584] req@000000001f9ec359 x1846780895154944/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233609 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233584.049831:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895154944/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233609 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233584.049836:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233584.049839:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233584.049847:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895154944:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233584.049852:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895155008:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233584.049885:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233584.049892:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233584/real 1761233584] req@00000000efa29eb9 x1846780895155008/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233609 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233584.049902:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895155008/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233609 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233584.049908:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233584.049910:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233584.049913:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895155008:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233584.049918:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895155072:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233584.049932:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233584.049938:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233584/real 1761233584] req@000000006d61dd06 x1846780895155072/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233609 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233584.049948:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895155072/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233609 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233584.049955:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233584.049956:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233584.049959:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895155072:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233584.049963:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895155136:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233584.049981:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233584.049987:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233584/real 1761233584] req@00000000ef655201 x1846780895155136/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233609 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233584.049996:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895155136/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233609 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233584.050003:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233584.050005:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233584.050007:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895155136:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233584.050011:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895155200:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233584.050026:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233584.050031:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233584/real 1761233584] req@000000004130b6bd x1846780895155200/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233609 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233584.050040:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895155200/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233609 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233584.050047:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233584.050049:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233584.050051:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895155200:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233584.817485:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233585.841388:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233586.865424:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233587.889445:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233588.913466:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233589.169337:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233589.169346:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233589.169350:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233589.169360:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6150 00000100:00080000:0.0:1761233589.169363:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233589.169369:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233589.169400:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895155264/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233589.169424:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233589.169427:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233589.169428:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233589.169430:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6150 00000100:00080000:0.0:1761233589.169432:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233589.169434:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233589.169441:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233589.169448:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895155328/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233589.169453:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233589.169456:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233589.169457:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233589.169460:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6150 00000100:00080000:0.0:1761233589.169461:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233589.169463:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233589.169466:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233589.169470:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895155392/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233589.169475:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233589.169477:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233589.169478:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233589.169480:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6150 00000100:00080000:0.0:1761233589.169481:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233589.169483:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233589.169485:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233589.169490:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895155456/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233589.169495:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233589.169497:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233589.169498:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233589.169500:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6150 00000100:00080000:0.0:1761233589.169501:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233589.169503:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233589.169506:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233589.169510:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895155520/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233589.169514:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895155264:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233589.169527:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233589.169530:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233589.169530:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233589.169534:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6150 00000100:00080000:0.0:1761233589.169535:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233589.169537:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233589.169540:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233589.169570:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895155584/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233589.169576:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233589.169578:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233589.169579:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233589.169581:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6150 00000100:00080000:0.0:1761233589.169583:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233589.169585:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233589.169588:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233589.169596:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895155648/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761233589.169602:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233589.169625:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233589/real 1761233589] req@000000009d1c7f73 x1846780895155264/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233614 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233589.169638:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895155264/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233614 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233589.169645:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233589.169647:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233589.169650:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895155264:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233589.169655:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895155328:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233589.169669:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233589.169674:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233589/real 1761233589] req@00000000d67bb589 x1846780895155328/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233614 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233589.169680:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895155328/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233614 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233589.169685:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233589.169686:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233589.169687:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895155328:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233589.169691:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895155392:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233589.169701:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233589.169705:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233589/real 1761233589] req@00000000a4a2c44a x1846780895155392/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233614 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233589.169711:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895155392/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233614 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233589.169715:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233589.169717:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233589.169719:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895155392:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233589.169722:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895155456:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233589.169731:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233589.169735:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233589/real 1761233589] req@000000004a74da19 x1846780895155456/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233614 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233589.169742:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895155456/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233614 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233589.169749:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233589.169751:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233589.169753:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895155456:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233589.169783:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895155520:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233589.169808:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233589.169818:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233589/real 1761233589] req@000000002d1b8ef7 x1846780895155520/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233614 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233589.169830:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895155520/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233614 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233589.169839:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233589.169842:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233589.169845:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895155520:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233589.169851:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895155584:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233589.169874:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233589.169882:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233589/real 1761233589] req@000000007bd1ec64 x1846780895155584/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233614 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233589.169895:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895155584/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233614 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233589.169902:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233589.169904:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233589.169907:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895155584:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233589.169913:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895155648:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233589.169932:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233589.169939:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233589/real 1761233589] req@000000001db96463 x1846780895155648/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233614 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233589.169950:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895155648/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233614 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233589.169958:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233589.169960:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233589.169962:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895155648:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233589.937453:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233590.961489:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233591.985518:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233593.009442:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233594.033377:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233594.289350:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233594.289360:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233594.289364:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233594.289376:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6155 00000100:00080000:0.0:1761233594.289380:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233594.289390:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233594.289426:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895155712/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233594.289451:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233594.289455:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233594.289456:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233594.289458:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6155 00000100:00080000:0.0:1761233594.289461:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233594.289463:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233594.289493:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233594.289521:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895155776/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233594.289529:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233594.289532:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233594.289534:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233594.289538:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6155 00000100:00080000:0.0:1761233594.289540:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233594.289544:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233594.289548:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233594.289558:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895155840/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233594.289566:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233594.289571:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1761233594.289571:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895155712:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233594.289574:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233594.289583:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6155 00000100:00080000:0.0:1761233594.289587:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233594.289589:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233594.289593:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233594.289601:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895155904/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233594.289608:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233594.289611:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233594.289622:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233594.289625:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6155 00000100:00080000:0.0:1761233594.289627:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233594.289630:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233594.289634:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233594.289644:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895155968/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233594.289650:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233594.289653:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233594.289654:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233594.289656:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6155 00000100:00080000:0.0:1761233594.289658:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233594.289661:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233594.289663:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233594.289671:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895156032/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233594.289683:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233594.289690:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233594.289691:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1761233594.289695:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233594.289699:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6155 00000100:00080000:0.0:1761233594.289702:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233594.289705:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233594.289709:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233594.289718:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895156096/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233594.289735:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233594/real 1761233594] req@00000000f3ed2588 x1846780895155712/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233619 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233594.289759:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f3ed2588 x1846780895155712/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233619 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233594.289773:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233594.289778:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233594.289783:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895155712:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233594.289794:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895155776:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233594.289829:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233594.289841:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233594/real 1761233594] req@000000005d83493f x1846780895155776/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233619 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233594.289858:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005d83493f x1846780895155776/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233619 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233594.289869:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233594.289872:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233594.289876:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895155776:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233594.289908:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895155840:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233594.289939:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233594.289951:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233594/real 1761233594] req@000000001db96463 x1846780895155840/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233619 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233594.289966:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895155840/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233619 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233594.289984:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233594.289987:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233594.289990:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895155840:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233594.289997:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895155904:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233594.290020:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233594.290031:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233594/real 1761233594] req@000000007bd1ec64 x1846780895155904/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233619 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233594.290046:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895155904/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233619 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233594.290056:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233594.290058:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233594.290062:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895155904:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233594.290070:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895155968:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233594.290114:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233594.290125:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233594/real 1761233594] req@000000002d1b8ef7 x1846780895155968/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233619 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233594.290137:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895155968/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233619 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233594.290147:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233594.290149:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233594.290153:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895155968:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233594.290160:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895156032:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233594.290185:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233594.290195:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233594/real 1761233594] req@000000004a74da19 x1846780895156032/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233619 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233594.290208:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895156032/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233619 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233594.290251:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233594.290264:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233594.290269:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895156032:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233594.290276:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895156096:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233594.290303:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233594.290314:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233594/real 1761233594] req@00000000a4a2c44a x1846780895156096/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233619 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233594.290339:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895156096/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233619 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233594.290349:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233594.290351:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233594.290355:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895156096:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233595.057421:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233596.081367:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233597.105532:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233598.129375:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233599.153415:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233599.409328:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233599.409341:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233599.409344:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233599.409356:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6160 00000100:00080000:0.0:1761233599.409360:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233599.409367:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233599.409428:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895156160/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233599.409449:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233599.409453:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233599.409454:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233599.409458:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6160 00000100:00080000:0.0:1761233599.409460:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233599.409464:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233599.409475:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233599.409485:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895156160:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233599.409486:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895156224/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233599.409495:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233599.409498:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233599.409499:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233599.409502:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6160 00000100:00080000:0.0:1761233599.409504:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233599.409507:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233599.409511:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233599.409518:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895156288/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233599.409524:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233599.409527:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233599.409528:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233599.409531:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6160 00000100:00080000:0.0:1761233599.409533:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233599.409535:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233599.409539:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233599.409545:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895156352/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761233599.409549:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233599.409553:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233599.409555:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233599.409556:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233599.409560:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6160 00000100:00080000:0.0:1761233599.409562:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233599.409564:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233599.409568:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233599.409571:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233599/real 1761233599] req@00000000d67bb589 x1846780895156160/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233624 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233599.409575:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895156416/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233599.409583:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233599.409586:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1761233599.409586:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895156160/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233624 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233599.409587:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233599.409591:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6160 00000100:00080000:0.0:1761233599.409593:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233599.409596:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:1.0:1761233599.409596:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233599.409599:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000008:00100000:0.0:1761233599.409600:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233599.409603:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895156160:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233599.409607:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895156480/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233599.409613:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233599.409615:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233599.409616:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233599.409619:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6160 00000100:00080000:0.0:1761233599.409621:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761233599.409622:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895156224:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:0.0:1761233599.409624:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233599.409627:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233599.409634:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895156544/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761233599.409642:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233599.409650:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233599/real 1761233599] req@000000009d1c7f73 x1846780895156224/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233624 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233599.409661:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895156224/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233624 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233599.409668:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233599.409670:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233599.409673:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895156224:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233599.409679:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895156288:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233599.409698:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233599.409704:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233599/real 1761233599] req@000000004130b6bd x1846780895156288/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233624 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233599.409710:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895156288/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233624 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233599.409719:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233599.409720:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233599.409722:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895156288:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233599.409725:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895156352:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233599.409735:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233599.409738:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233599/real 1761233599] req@00000000ef655201 x1846780895156352/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233624 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233599.409744:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895156352/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233624 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233599.409748:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233599.409750:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233599.409751:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895156352:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233599.409755:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895156416:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233599.409764:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233599.409768:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233599/real 1761233599] req@000000006d61dd06 x1846780895156416/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233624 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233599.409774:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895156416/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233624 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233599.409778:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233599.409779:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233599.409781:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895156416:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233599.409784:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895156480:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233599.409793:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233599.409797:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233599/real 1761233599] req@00000000efa29eb9 x1846780895156480/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233624 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233599.409802:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895156480/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233624 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233599.409806:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233599.409808:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233599.409809:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895156480:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233599.409823:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895156544:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233599.409832:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233599.409835:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233599/real 1761233599] req@000000001f9ec359 x1846780895156544/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233624 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233599.409841:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895156544/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233624 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233599.409846:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233599.409847:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233599.409848:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895156544:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233600.177495:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233601.201421:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233602.225521:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233603.249558:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233604.273464:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233604.529369:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233604.529381:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233604.529386:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233604.529398:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6165 00000100:00080000:2.0:1761233604.529402:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233604.529409:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233604.529455:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895156608/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233604.529477:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233604.529480:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233604.529482:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233604.529484:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6165 00000100:00080000:2.0:1761233604.529486:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233604.529488:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233604.529497:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233604.529506:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895156672/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233604.529513:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233604.529515:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233604.529516:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761233604.529520:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895156608:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761233604.529521:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6165 00000100:00080000:2.0:1761233604.529523:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233604.529527:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233604.529534:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233604.529542:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895156736/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233604.529548:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233604.529550:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233604.529552:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233604.529554:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6165 00000100:00080000:2.0:1761233604.529556:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233604.529558:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233604.529561:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233604.529568:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895156800/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233604.529574:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233604.529576:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233604.529577:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233604.529579:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6165 00000100:00080000:2.0:1761233604.529580:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233604.529583:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233604.529585:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233604.529591:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895156864/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233604.529596:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233604.529598:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233604.529598:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233604.529601:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6165 00000100:00080000:2.0:1761233604.529603:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233604.529606:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233604.529608:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233604.529615:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895156928/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233604.529620:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1761233604.529621:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233604.529623:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233604.529625:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233604.529629:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6165 00000100:00080000:2.0:1761233604.529630:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233604.529633:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233604.529635:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233604.529642:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895156992/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233604.529650:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233604/real 1761233604] req@0000000064a2cea0 x1846780895156608/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233629 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233604.529666:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895156608/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233629 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233604.529675:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233604.529677:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233604.529681:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895156608:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761233604.529710:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895156672:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233604.529748:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233604.529757:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233604/real 1761233604] req@000000007091f547 x1846780895156672/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233629 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233604.529770:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895156672/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233629 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233604.529778:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233604.529780:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233604.529784:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895156672:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233604.529790:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895156736:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233604.529812:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233604.529819:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233604/real 1761233604] req@000000007dcf84ae x1846780895156736/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233629 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233604.529829:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895156736/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233629 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233604.529838:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233604.529840:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233604.529843:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895156736:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233604.529849:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895156800:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233604.529867:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233604.529874:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233604/real 1761233604] req@0000000040d5f361 x1846780895156800/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233629 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233604.529885:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895156800/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233629 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233604.529892:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233604.529894:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233604.529897:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895156800:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233604.529902:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895156864:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233604.529920:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233604.529927:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233604/real 1761233604] req@000000009d315fa5 x1846780895156864/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233629 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233604.529938:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895156864/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233629 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233604.529945:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233604.529947:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233604.529950:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895156864:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233604.529956:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895156928:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233604.529974:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233604.529982:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233604/real 1761233604] req@0000000033765045 x1846780895156928/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233629 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233604.530004:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895156928/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233629 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233604.530012:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233604.530014:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233604.530017:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895156928:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233604.530022:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895156992:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233604.530040:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233604.530047:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233604/real 1761233604] req@000000004f29ca26 x1846780895156992/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233629 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233604.530058:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895156992/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233629 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233604.530065:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233604.530067:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233604.530070:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895156992:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233605.297603:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233606.321704:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233607.345453:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233608.369632:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233609.393478:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233609.649495:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233609.649518:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233609.649526:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233609.649546:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6170 00000100:00080000:2.0:1761233609.649551:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233609.649562:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233609.649624:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895157056/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233609.649665:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233609.649671:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233609.649673:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233609.649678:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6170 00000100:00080000:2.0:1761233609.649681:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233609.649685:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233609.649697:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233609.649711:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895157120/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233609.649726:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233609.649731:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233609.649734:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233609.649738:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6170 00000100:00080000:2.0:1761233609.649740:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233609.649744:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233609.649750:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233609.649764:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895157184/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233609.649773:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895157056:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761233609.649780:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233609.649792:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233609.649794:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233609.649800:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6170 00000100:00080000:2.0:1761233609.649802:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233609.649806:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233609.649811:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233609.649826:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895157248/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233609.649837:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233609.649841:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233609.649844:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233609.649860:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6170 00000100:00080000:2.0:1761233609.649862:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233609.649866:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233609.649871:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233609.649883:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895157312/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233609.649891:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233609.649894:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233609.649896:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233609.649901:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6170 00000100:00080000:2.0:1761233609.649904:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233609.649908:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233609.649913:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233609.649926:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895157376/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233609.649935:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233609.649939:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233609.649941:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233609.649945:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6170 00000100:00080000:2.0:1761233609.649948:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233609.649954:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233609.649959:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233609.649969:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895157120:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233609.649975:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895157440/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233609.650073:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895157184:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233609.650136:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895157248:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233609.650167:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895157312:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233609.650195:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895157376:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233609.650308:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895157440:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233610.417514:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233611.441433:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233612.465442:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1761233612.721612:0:2208:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:1.0:1761233612.721661:0:2208:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1761233612.721677:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233612.721717:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233612.721724:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233612.721729:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233612.721735:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233612.721740:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233612.721747:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:0.0:1761233612.721789:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233609/real 1761233612] req@000000006ff1fdbc x1846780895157184/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233634 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233612.721821:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895157184/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233634 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233612.721836:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233612.721841:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233612.721847:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895157184:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233612.721856:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233609/real 1761233612] req@0000000088939481 x1846780895157248/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233634 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233612.721871:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895157248/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233634 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233612.721881:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233612.721885:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233612.721888:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895157248:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233612.721896:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233609/real 1761233612] req@000000006d8c3eb0 x1846780895157312/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233634 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233612.721910:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895157312/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233634 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233612.721920:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233612.721923:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233612.721926:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895157312:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233612.721938:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233609/real 1761233612] req@000000006c1041ba x1846780895157376/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233634 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233612.721952:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895157376/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233634 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233612.721961:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233612.721964:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233612.721969:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895157376:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233612.721975:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233609/real 1761233612] req@00000000fd4aa3a2 x1846780895157440/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233634 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233612.721990:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895157440/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233634 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233612.722000:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233612.722003:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233612.722007:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895157440:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761233612.722014:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233609/real 1761233612] req@00000000fd3935a6 x1846780895157056/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233634 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233612.722029:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895157056/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233634 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233612.722038:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233612.722041:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233612.722045:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895157056:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761233612.722074:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233609/real 1761233612] req@00000000135e9a6e x1846780895157120/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233634 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233612.722089:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895157120/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233634 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233612.722096:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233612.722099:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233612.722103:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895157120:192.168.201.157@tcp:38:kworker/u8:3.0 00010000:00010000:3.0:1761233613.489457:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233614.513541:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233614.769351:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233614.769366:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233614.769372:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233614.769386:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6175 00000100:00080000:2.0:1761233614.769390:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233614.769398:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233614.769444:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895157504/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233614.769490:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233614.769493:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233614.769495:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233614.769498:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6175 00000100:00080000:2.0:1761233614.769500:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233614.769502:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233614.769513:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233614.769523:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895157568/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233614.769531:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233614.769534:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233614.769537:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233614.769540:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6175 00000100:00080000:2.0:1761233614.769542:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233614.769545:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233614.769549:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233614.769563:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895157504:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761233614.769563:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895157632/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233614.769583:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233614.769585:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233614.769586:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233614.769592:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6175 00000100:00080000:2.0:1761233614.769594:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233614.769598:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233614.769602:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233614.769613:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895157696/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233614.769621:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233614.769625:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233614.769625:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233614.769628:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6175 00000100:00080000:2.0:1761233614.769630:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233614.769633:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233614.769636:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233614.769643:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895157760/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233614.769649:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233614.769669:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233614.769671:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233614.769675:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6175 00000100:00080000:2.0:1761233614.769678:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233614.769683:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233614.769688:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761233614.769698:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233614.769701:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895157824/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233614.769711:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233614.769715:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233614.769717:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233614.769722:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6175 00000100:00080000:2.0:1761233614.769725:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233614.769729:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233614.769734:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233614.769735:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233614/real 1761233614] req@00000000f4e415d3 x1846780895157504/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233639 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233614.769747:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895157888/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233614.769760:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895157504/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233639 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233614.769782:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233614.769787:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233614.769792:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895157504:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761233614.769801:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895157568:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233614.769831:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233614.769842:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233614/real 1761233614] req@0000000099b7e5d9 x1846780895157568/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233639 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233614.769855:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895157568/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233639 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233614.769864:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233614.769866:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233614.769870:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895157568:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233614.769906:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895157632:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233614.769932:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233614.769942:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233614/real 1761233614] req@00000000c10cebe6 x1846780895157632/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233639 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233614.769955:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895157632/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233639 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233614.769964:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233614.769967:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233614.769970:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895157632:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233614.769978:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895157696:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233614.770002:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233614.770011:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233614/real 1761233614] req@00000000a1e8c6bb x1846780895157696/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233639 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233614.770024:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895157696/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233639 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233614.770033:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233614.770035:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233614.770038:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895157696:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233614.770045:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895157760:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233614.770068:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233614.770077:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233614/real 1761233614] req@000000008ebccab5 x1846780895157760/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233639 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233614.770097:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895157760/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233639 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233614.770107:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233614.770110:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233614.770113:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895157760:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233614.770121:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895157824:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233614.770165:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233614.770175:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233614/real 1761233614] req@000000004ae5bc9b x1846780895157824/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233639 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233614.770189:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895157824/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233639 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233614.770198:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233614.770267:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233614.770273:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895157824:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233614.770281:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895157888:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233614.770305:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233614.770314:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233614/real 1761233614] req@00000000ac7d8d5b x1846780895157888/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233639 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233614.770326:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895157888/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233639 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233614.770335:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233614.770337:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233614.770340:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895157888:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233615.537390:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233616.561428:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233617.585363:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233618.609397:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233619.633399:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233619.889346:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233619.889354:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233619.889357:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233619.889367:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6180 00000100:00080000:0.0:1761233619.889370:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233619.889376:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233619.889408:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895157952/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233619.889433:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233619.889436:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233619.889437:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233619.889439:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6180 00000100:00080000:0.0:1761233619.889441:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233619.889443:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233619.889451:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233619.889458:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895158016/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233619.889463:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233619.889466:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233619.889467:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233619.889469:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6180 00000100:00080000:0.0:1761233619.889471:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233619.889472:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233619.889475:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233619.889479:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895158080/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233619.889484:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233619.889486:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233619.889486:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233619.889489:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6180 00000100:00080000:0.0:1761233619.889490:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233619.889492:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233619.889495:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233619.889500:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895158144/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233619.889505:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233619.889507:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233619.889508:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233619.889511:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6180 00000100:00080000:0.0:1761233619.889513:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761233619.889514:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895157952:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233619.889519:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233619.889524:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233619.889529:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895158208/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233619.889533:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233619.889535:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233619.889536:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233619.889539:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6180 00000100:00080000:0.0:1761233619.889540:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233619.889542:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233619.889545:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233619.889552:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895158272/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233619.889557:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233619.889559:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233619.889560:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233619.889561:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6180 00000100:00080000:0.0:1761233619.889563:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233619.889565:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233619.889568:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233619.889573:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895158336/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761233619.889588:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233619.889607:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233619/real 1761233619] req@000000005593c994 x1846780895157952/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233644 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233619.889619:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895157952/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233644 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233619.889627:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233619.889629:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233619.889632:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895157952:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761233619.889637:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895158016:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233619.889650:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233619.889655:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233619/real 1761233619] req@00000000c6e06344 x1846780895158016/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233644 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233619.889661:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895158016/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233644 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233619.889666:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233619.889667:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233619.889668:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895158016:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233619.889672:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895158080:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233619.889682:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233619.889686:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233619/real 1761233619] req@000000004862b808 x1846780895158080/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233644 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233619.889692:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895158080/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233644 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233619.889696:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233619.889697:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233619.889699:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895158080:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233619.889719:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895158144:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233619.889729:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233619.889733:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233619/real 1761233619] req@000000006f2efc8b x1846780895158144/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233644 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233619.889739:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895158144/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233644 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233619.889744:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233619.889745:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233619.889746:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895158144:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233619.889749:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895158208:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233619.889758:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233619.889762:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233619/real 1761233619] req@00000000eca9ddc2 x1846780895158208/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233644 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233619.889768:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895158208/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233644 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233619.889773:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233619.889774:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233619.889775:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895158208:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233619.889778:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895158272:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233619.889787:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233619.889790:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233619/real 1761233619] req@0000000034d49f59 x1846780895158272/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233644 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233619.889797:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895158272/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233644 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233619.889801:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233619.889802:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233619.889804:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895158272:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233619.889807:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895158336:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233619.889816:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233619.889819:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233619/real 1761233619] req@000000005beaae31 x1846780895158336/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233644 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233619.889825:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895158336/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233644 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233619.889829:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233619.889830:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233619.889832:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895158336:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233620.657423:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233621.681368:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233622.705431:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233623.729551:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233624.753510:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233625.009393:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233625.009410:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233625.009417:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233625.009435:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6185 00000100:00080000:1.0:1761233625.009441:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233625.009453:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233625.009529:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cb5be845 x1846780895158400/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233625.009580:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233625.009585:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233625.009587:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233625.009592:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6185 00000100:00080000:1.0:1761233625.009595:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233625.009600:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233625.009613:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233625.009626:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000880f16b4 x1846780895158464/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233625.009637:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233625.009642:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233625.009644:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233625.009649:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6185 00000100:00080000:1.0:1761233625.009651:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233625.009655:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233625.009659:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233625.009669:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000038209f5 x1846780895158528/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233625.009680:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233625.009684:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233625.009685:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233625.009690:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6185 00000100:00080000:1.0:1761233625.009693:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233625.009698:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233625.009704:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233625.009715:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d66930c2 x1846780895158592/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233625.009725:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233625.009728:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233625.009730:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233625.009737:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6185 00000100:00080000:1.0:1761233625.009740:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233625.009744:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233625.009752:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233625.009768:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895158400:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233625.009771:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ffe86d04 x1846780895158656/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233625.009802:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233625.009805:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233625.009807:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233625.009813:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6185 00000100:00080000:1.0:1761233625.009815:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233625.009820:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233625.009828:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233625.009846:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df5b2270 x1846780895158720/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233625.009863:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233625.009867:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233625.009869:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233625.009873:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6185 00000100:00080000:1.0:1761233625.009875:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233625.009878:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233625.009883:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233625.009895:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df0b94a8 x1846780895158784/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:2.0:1761233625.009912:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233625.009958:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233625/real 1761233625] req@00000000cb5be845 x1846780895158400/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233650 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233625.010008:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cb5be845 x1846780895158400/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233650 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233625.010041:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233625.010045:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233625.010050:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895158400:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761233625.010061:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895158464:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761233625.010090:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233625.010100:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233625/real 1761233625] req@00000000880f16b4 x1846780895158464/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233650 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233625.010112:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000880f16b4 x1846780895158464/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233650 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233625.010120:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233625.010123:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233625.010126:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895158464:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233625.010133:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895158528:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233625.010151:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233625.010158:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233625/real 1761233625] req@00000000038209f5 x1846780895158528/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233650 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233625.010169:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000038209f5 x1846780895158528/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233650 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233625.010177:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233625.010179:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233625.010182:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895158528:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233625.010189:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895158592:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233625.010207:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233625.010290:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233625/real 1761233625] req@00000000d66930c2 x1846780895158592/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233650 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233625.010304:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d66930c2 x1846780895158592/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233650 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233625.010311:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233625.010313:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233625.010316:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895158592:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233625.010354:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895158656:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761233625.010377:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233625.010385:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233625/real 1761233625] req@00000000ffe86d04 x1846780895158656/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233650 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233625.010394:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ffe86d04 x1846780895158656/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233650 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233625.010402:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233625.010404:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233625.010407:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895158656:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233625.010412:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895158720:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233625.010426:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233625.010432:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233625/real 1761233625] req@00000000df5b2270 x1846780895158720/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233650 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233625.010442:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df5b2270 x1846780895158720/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233650 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233625.010449:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233625.010451:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233625.010453:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895158720:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233625.010458:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895158784:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233625.010472:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233625.010478:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233625/real 1761233625] req@00000000df0b94a8 x1846780895158784/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233650 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233625.010487:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df0b94a8 x1846780895158784/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233650 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233625.010493:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233625.010495:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233625.010498:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895158784:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233625.777601:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233626.801435:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233627.825520:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233628.849528:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233629.873397:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233630.129399:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233630.129413:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233630.129418:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233630.129433:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6191 00000100:00080000:1.0:1761233630.129438:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233630.129447:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233630.129506:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac65ecb7 x1846780895158848/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233630.129534:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233630.129538:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233630.129540:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233630.129544:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6191 00000100:00080000:1.0:1761233630.129546:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233630.129552:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233630.129564:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233630.129579:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006a5e9865 x1846780895158912/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233630.129587:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233630.129590:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233630.129592:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233630.129595:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6191 00000100:00080000:1.0:1761233630.129598:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233630.129602:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233630.129607:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233630.129619:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004361f727 x1846780895158976/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233630.129628:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233630.129632:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233630.129633:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233630.129637:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6191 00000100:00080000:1.0:1761233630.129640:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233630.129644:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233630.129649:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233630.129664:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df0b94a8 x1846780895159040/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761233630.129667:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac65ecb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895158848:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233630.129676:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233630.129683:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233630.129686:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233630.129690:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6191 00000100:00080000:1.0:1761233630.129693:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233630.129696:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233630.129701:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233630.129718:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df5b2270 x1846780895159104/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233630.129727:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233630.129731:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233630.129733:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233630.129737:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6191 00000100:00080000:1.0:1761233630.129739:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233630.129743:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233630.129750:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233630.129763:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ffe86d04 x1846780895159168/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233630.129772:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233630.129776:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233630.129778:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1761233630.129781:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:1.0:1761233630.129786:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6191 00000100:00080000:1.0:1761233630.129790:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233630.129794:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233630.129799:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233630.129820:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233630/real 1761233630] req@00000000ac65ecb7 x1846780895158848/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233655 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233630.129821:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d66930c2 x1846780895159232/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761233630.129921:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac65ecb7 x1846780895158848/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233655 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233630.129940:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233630.129945:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233630.129950:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac65ecb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895158848:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761233630.129959:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a5e9865 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895158912:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761233630.129986:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233630.129995:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233630/real 1761233630] req@000000006a5e9865 x1846780895158912/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233655 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233630.130007:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006a5e9865 x1846780895158912/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233655 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233630.130015:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233630.130018:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233630.130021:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a5e9865 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895158912:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233630.130027:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004361f727 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895158976:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233630.130044:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233630.130050:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233630/real 1761233630] req@000000004361f727 x1846780895158976/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233655 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233630.130061:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004361f727 x1846780895158976/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233655 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233630.130068:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233630.130070:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233630.130072:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004361f727 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895158976:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233630.130107:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895159040:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233630.130127:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233630.130134:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233630/real 1761233630] req@00000000df0b94a8 x1846780895159040/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233655 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233630.130144:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df0b94a8 x1846780895159040/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233655 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233630.130151:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233630.130154:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233630.130167:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895159040:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233630.130173:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895159104:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761233630.130188:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233630.130194:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233630/real 1761233630] req@00000000df5b2270 x1846780895159104/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233655 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233630.130251:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df5b2270 x1846780895159104/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233655 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233630.130259:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233630.130262:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233630.130264:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895159104:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233630.130271:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895159168:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233630.130292:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233630.130300:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233630/real 1761233630] req@00000000ffe86d04 x1846780895159168/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233655 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233630.130311:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ffe86d04 x1846780895159168/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233655 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233630.130318:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233630.130321:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233630.130323:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895159168:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233630.130329:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895159232:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233630.130345:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233630.130351:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233630/real 1761233630] req@00000000d66930c2 x1846780895159232/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233655 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233630.130361:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d66930c2 x1846780895159232/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233655 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233630.130368:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233630.130370:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233630.130372:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895159232:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233630.897372:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233631.921463:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233632.945397:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233633.969356:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233634.993402:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233635.249318:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233635.249328:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233635.249332:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233635.249344:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6196 00000100:00080000:1.0:1761233635.249348:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233635.249354:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233635.249385:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000038209f5 x1846780895159296/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233635.249408:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233635.249411:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233635.249412:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233635.249414:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6196 00000100:00080000:1.0:1761233635.249416:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233635.249418:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233635.249425:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233635.249431:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000880f16b4 x1846780895159360/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233635.249437:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233635.249439:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233635.249440:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233635.249442:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6196 00000100:00080000:1.0:1761233635.249445:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233635.249447:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233635.249450:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233635.249455:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cb5be845 x1846780895159424/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233635.249460:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233635.249462:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233635.249464:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1761233635.249467:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895159296:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:1.0:1761233635.249468:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6196 00000100:00080000:1.0:1761233635.249471:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233635.249475:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233635.249478:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233635.249484:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a95774fb x1846780895159488/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233635.249490:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233635.249492:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233635.249493:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233635.249495:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6196 00000100:00080000:1.0:1761233635.249496:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233635.249499:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233635.249502:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233635.249507:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000c736bb2 x1846780895159552/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233635.249512:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233635.249515:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233635.249516:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233635.249518:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6196 00000100:00080000:1.0:1761233635.249520:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233635.249522:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233635.249524:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1761233635.249530:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233635.249533:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090ac513e x1846780895159616/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233635.249538:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233635.249541:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233635.249542:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233635.249544:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6196 00000100:00080000:1.0:1761233635.249546:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233635.249549:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233635.249552:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233635.249555:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233635/real 1761233635] req@00000000038209f5 x1846780895159296/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233660 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233635.249559:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000053e52331 x1846780895159680/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761233635.249582:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000038209f5 x1846780895159296/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233660 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233635.249591:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233635.249593:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233635.249597:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895159296:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761233635.249603:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895159360:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233635.249617:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233635.249621:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233635/real 1761233635] req@00000000880f16b4 x1846780895159360/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233660 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233635.249628:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000880f16b4 x1846780895159360/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233660 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233635.249632:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233635.249634:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233635.249635:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895159360:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233635.249654:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895159424:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233635.249665:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233635.249669:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233635/real 1761233635] req@00000000cb5be845 x1846780895159424/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233660 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233635.249675:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cb5be845 x1846780895159424/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233660 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233635.249680:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233635.249681:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233635.249682:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895159424:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233635.249685:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895159488:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233635.249694:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233635.249698:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233635/real 1761233635] req@00000000a95774fb x1846780895159488/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233660 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233635.249705:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a95774fb x1846780895159488/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233660 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233635.249709:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233635.249710:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233635.249711:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895159488:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233635.249714:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895159552:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233635.249724:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233635.249727:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233635/real 1761233635] req@000000000c736bb2 x1846780895159552/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233660 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233635.249733:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000000c736bb2 x1846780895159552/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233660 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233635.249737:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233635.249738:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233635.249739:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895159552:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233635.249742:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895159616:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233635.249761:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233635.249766:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233635/real 1761233635] req@0000000090ac513e x1846780895159616/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233660 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233635.249772:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090ac513e x1846780895159616/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233660 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233635.249780:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233635.249781:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233635.249783:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895159616:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233635.249787:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895159680:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233635.249797:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233635.249802:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233635/real 1761233635] req@0000000053e52331 x1846780895159680/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233660 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233635.249808:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000053e52331 x1846780895159680/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233660 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233635.249812:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233635.249813:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233635.249814:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895159680:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233636.017409:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233637.041375:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233638.065514:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233639.089446:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233640.113576:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233640.369326:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233640.369336:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233640.369339:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233640.369349:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6201 00000100:00080000:1.0:1761233640.369352:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233640.369358:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233640.369395:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000568f2a96 x1846780895159744/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233640.369416:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233640.369419:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233640.369420:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233640.369423:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6201 00000100:00080000:1.0:1761233640.369424:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233640.369426:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233640.369434:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233640.369442:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c652d0ad x1846780895159808/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761233640.369449:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895159744:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:1.0:1761233640.369450:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233640.369457:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233640.369458:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233640.369461:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6201 00000100:00080000:1.0:1761233640.369463:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233640.369465:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233640.369468:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233640.369477:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ba3340f x1846780895159872/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233640.369481:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233640.369484:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233640.369485:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233640.369487:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6201 00000100:00080000:1.0:1761233640.369489:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233640.369491:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233640.369495:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233640.369501:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d74b2e3c x1846780895159936/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233640.369506:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233640.369508:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233640.369509:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233640.369511:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6201 00000100:00080000:1.0:1761233640.369512:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233640.369516:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:2.0:1761233640.369518:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:1.0:1761233640.369521:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233640.369530:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001aedf9b1 x1846780895160000/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233640.369536:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233640.369538:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233640.369539:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1761233640.369540:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233640/real 1761233640] req@00000000568f2a96 x1846780895159744/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233665 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233640.369544:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6201 00000100:00080000:1.0:1761233640.369547:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233640.369552:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233640.369558:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233640.369565:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000568f2a96 x1846780895159744/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233665 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233640.369566:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a611ee34 x1846780895160064/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233640.369595:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233640.369597:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233640.369598:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233640.369601:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233640.369601:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233640.369608:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895159744:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:1.0:1761233640.369609:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6201 00000100:00080000:1.0:1761233640.369613:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233640.369616:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233640.369619:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233640.369625:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000328368c8 x1846780895160128/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761233640.369634:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895159808:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233640.369653:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233640.369658:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233640/real 1761233640] req@00000000c652d0ad x1846780895159808/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233665 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233640.369665:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c652d0ad x1846780895159808/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233665 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233640.369670:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233640.369672:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233640.369674:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895159808:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233640.369677:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895159872:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233640.369687:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233640.369690:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233640/real 1761233640] req@000000007ba3340f x1846780895159872/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233665 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233640.369696:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007ba3340f x1846780895159872/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233665 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233640.369701:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233640.369702:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233640.369703:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895159872:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233640.369706:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895159936:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233640.369715:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233640.369718:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233640/real 1761233640] req@00000000d74b2e3c x1846780895159936/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233665 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233640.369724:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d74b2e3c x1846780895159936/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233665 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233640.369729:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233640.369730:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233640.369732:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895159936:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233640.369735:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895160000:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233640.369768:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233640.369773:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233640/real 1761233640] req@000000001aedf9b1 x1846780895160000/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233665 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233640.369779:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001aedf9b1 x1846780895160000/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233665 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233640.369783:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233640.369785:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233640.369787:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895160000:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233640.369790:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895160064:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233640.369800:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233640.369804:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233640/real 1761233640] req@00000000a611ee34 x1846780895160064/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233665 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233640.369810:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a611ee34 x1846780895160064/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233665 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233640.369814:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233640.369815:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233640.369817:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895160064:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233640.369820:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895160128:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233640.369829:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233640.369833:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233640/real 1761233640] req@00000000328368c8 x1846780895160128/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233665 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233640.369839:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000328368c8 x1846780895160128/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233665 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233640.369843:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233640.369845:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233640.369846:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895160128:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233641.137414:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233642.161488:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233643.185481:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233644.209461:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233645.233467:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233645.489354:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233645.489377:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233645.489385:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233645.489398:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6206 00000100:00080000:1.0:1761233645.489402:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233645.489409:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233645.489449:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cbad49de x1846780895160192/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233645.489483:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233645.489487:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233645.489489:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233645.489493:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6206 00000100:00080000:1.0:1761233645.489495:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233645.489500:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233645.489510:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233645.489525:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e467e107 x1846780895160256/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761233645.489535:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbad49de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895160192:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:1.0:1761233645.489542:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233645.489547:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233645.489549:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233645.489554:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6206 00000100:00080000:1.0:1761233645.489557:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233645.489562:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233645.489568:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233645.489581:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090800dde x1846780895160320/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233645.489591:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233645.489596:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233645.489598:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233645.489601:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6206 00000100:00080000:1.0:1761233645.489603:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233645.489607:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233645.489613:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1761233645.489627:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233645.489630:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000328368c8 x1846780895160384/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233645.489643:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233645.489646:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233645.489649:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233645.489655:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6206 00000100:00080000:1.0:1761233645.489658:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1761233645.489680:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233645/real 1761233645] req@00000000cbad49de x1846780895160192/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233670 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233645.489699:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233645.489705:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233645.489705:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cbad49de x1846780895160192/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233670 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233645.489722:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1761233645.489723:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a611ee34 x1846780895160448/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233645.489731:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:1.0:1761233645.489742:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1761233645.489744:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbad49de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895160192:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:1.0:1761233645.489749:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233645.489752:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233645.489758:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6206 00000100:00080000:1.0:1761233645.489760:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233645.489767:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233645.489774:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233645.489786:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001aedf9b1 x1846780895160512/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233645.489795:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233645.489798:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233645.489800:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1761233645.489804:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e467e107 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895160256:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:1.0:1761233645.489807:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6206 00000100:00080000:1.0:1761233645.489814:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233645.489819:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233645.489824:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233645.489841:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d74b2e3c x1846780895160576/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:2.0:1761233645.489851:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233645.489871:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233645/real 1761233645] req@00000000e467e107 x1846780895160256/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233670 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233645.489892:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e467e107 x1846780895160256/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233670 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233645.489900:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233645.489903:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233645.489908:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e467e107 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895160256:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233645.489915:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090800dde pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895160320:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233645.489939:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233645.489948:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233645/real 1761233645] req@0000000090800dde x1846780895160320/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233670 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233645.489959:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090800dde x1846780895160320/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233670 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233645.489965:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233645.489968:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233645.489971:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090800dde pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895160320:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233645.489978:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895160384:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233645.489999:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233645.490007:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233645/real 1761233645] req@00000000328368c8 x1846780895160384/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233670 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233645.490015:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000328368c8 x1846780895160384/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233670 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233645.490020:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233645.490024:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233645.490027:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895160384:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233645.490033:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895160448:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233645.490056:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233645.490065:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233645/real 1761233645] req@00000000a611ee34 x1846780895160448/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233670 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233645.490076:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a611ee34 x1846780895160448/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233670 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233645.490085:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233645.490087:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233645.490091:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895160448:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233645.490097:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895160512:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233645.490113:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233645.490118:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233645/real 1761233645] req@000000001aedf9b1 x1846780895160512/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233670 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233645.490125:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001aedf9b1 x1846780895160512/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233670 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233645.490130:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233645.490132:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233645.490134:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895160512:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233645.490152:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895160576:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233645.490165:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233645.490169:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233645/real 1761233645] req@00000000d74b2e3c x1846780895160576/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233670 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233645.490176:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d74b2e3c x1846780895160576/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233670 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233645.490180:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233645.490182:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233645.490183:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895160576:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233646.257425:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233647.281449:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233648.305387:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233649.329424:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233650.353398:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233650.609304:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233650.609317:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233650.609321:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233650.609333:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6211 00000100:00080000:1.0:1761233650.609336:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233650.609390:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233650.609440:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ba3340f x1846780895160640/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233650.609461:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233650.609464:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233650.609465:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233650.609468:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6211 00000100:00080000:1.0:1761233650.609469:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233650.609472:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233650.609480:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233650.609508:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895160640:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761233650.609509:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c652d0ad x1846780895160704/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233650.609520:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233650.609522:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233650.609523:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233650.609525:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6211 00000100:00080000:1.0:1761233650.609527:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233650.609529:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233650.609532:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233650.609540:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000568f2a96 x1846780895160768/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233650.609546:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233650.609548:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233650.609549:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233650.609551:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6211 00000100:00080000:1.0:1761233650.609552:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233650.609553:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233650.609556:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233650.609561:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000053e52331 x1846780895160832/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233650.609566:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233650.609568:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233650.609569:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233650.609571:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6211 00000100:00080000:1.0:1761233650.609573:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233650.609577:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:2.0:1761233650.609592:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:1.0:1761233650.609598:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233650.609613:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090ac513e x1846780895160896/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761233650.609617:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233650/real 1761233650] req@000000007ba3340f x1846780895160640/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233675 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233650.609637:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233650.609641:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1761233650.609642:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007ba3340f x1846780895160640/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233675 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233650.609644:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233650.609652:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6211 00000100:00080000:1.0:1761233650.609655:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233650.609656:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233650.609660:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:2.0:1761233650.609661:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000008:00100000:1.0:1761233650.609667:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233650.609667:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895160640:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761233650.609680:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000c736bb2 x1846780895160960/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233650.609686:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233650.609689:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233650.609690:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1761233650.609694:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895160704:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:1.0:1761233650.609695:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6211 00000100:00080000:1.0:1761233650.609699:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233650.609703:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233650.609707:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233650.609717:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a95774fb x1846780895161024/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:2.0:1761233650.609724:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233650.609734:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233650/real 1761233650] req@00000000c652d0ad x1846780895160704/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233675 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233650.609742:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c652d0ad x1846780895160704/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233675 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233650.609748:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233650.609750:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233650.609752:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895160704:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233650.609755:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895160768:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233650.609766:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233650.609770:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233650/real 1761233650] req@00000000568f2a96 x1846780895160768/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233675 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233650.609777:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000568f2a96 x1846780895160768/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233675 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233650.609784:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233650.609786:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233650.609789:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895160768:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233650.609795:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895160832:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233650.609814:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233650.609822:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233650/real 1761233650] req@0000000053e52331 x1846780895160832/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233675 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233650.609834:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000053e52331 x1846780895160832/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233675 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233650.609841:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233650.609844:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233650.609847:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895160832:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233650.609853:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895160896:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233650.609874:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233650.609883:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233650/real 1761233650] req@0000000090ac513e x1846780895160896/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233675 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233650.609894:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090ac513e x1846780895160896/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233675 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233650.609902:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233650.609904:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233650.609907:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895160896:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233650.609913:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895160960:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233650.609933:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233650.609939:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233650/real 1761233650] req@000000000c736bb2 x1846780895160960/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233675 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233650.609946:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000000c736bb2 x1846780895160960/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233675 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233650.609951:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233650.609952:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233650.609954:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895160960:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233650.609970:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895161024:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233650.609981:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233650.609985:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233650/real 1761233650] req@00000000a95774fb x1846780895161024/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233675 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233650.609992:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a95774fb x1846780895161024/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233675 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233650.609996:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233650.609998:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233650.609999:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895161024:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233651.377383:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233652.401329:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233653.425397:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233654.449421:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233655.473414:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233655.729313:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233655.729323:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233655.729326:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233655.729336:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6216 00000100:00080000:1.0:1761233655.729338:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233655.729344:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233655.729375:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cb5be845 x1846780895161088/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233655.729397:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233655.729400:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233655.729401:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233655.729404:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6216 00000100:00080000:1.0:1761233655.729405:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233655.729407:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233655.729415:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233655.729424:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000880f16b4 x1846780895161152/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233655.729430:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233655.729433:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233655.729434:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233655.729437:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6216 00000100:00080000:1.0:1761233655.729438:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233655.729440:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233655.729443:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233655.729449:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895161088:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761233655.729450:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000038209f5 x1846780895161216/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233655.729462:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233655.729464:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233655.729465:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233655.729467:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6216 00000100:00080000:1.0:1761233655.729468:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233655.729470:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233655.729473:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233655.729481:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d66930c2 x1846780895161280/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233655.729486:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233655.729489:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233655.729490:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233655.729493:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6216 00000100:00080000:1.0:1761233655.729494:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233655.729496:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233655.729514:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233655.729521:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ffe86d04 x1846780895161344/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233655.729526:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233655.729528:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233655.729529:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233655.729531:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6216 00000100:00080000:1.0:1761233655.729533:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233655.729535:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233655.729537:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233655.729542:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df5b2270 x1846780895161408/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233655.729547:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233655.729549:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233655.729550:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233655.729552:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6216 00000100:00080000:1.0:1761233655.729554:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1761233655.729558:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:1.0:1761233655.729559:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233655.729564:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233655.729571:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df0b94a8 x1846780895161472/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761233655.729585:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233655/real 1761233655] req@00000000cb5be845 x1846780895161088/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233680 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233655.729604:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cb5be845 x1846780895161088/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233680 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233655.729614:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233655.729617:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233655.729621:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895161088:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761233655.729628:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895161152:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233655.729655:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233655.729664:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233655/real 1761233655] req@00000000880f16b4 x1846780895161152/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233680 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233655.729676:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000880f16b4 x1846780895161152/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233680 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233655.729684:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233655.729687:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233655.729690:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895161152:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233655.729715:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895161216:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233655.729731:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233655.729737:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233655/real 1761233655] req@00000000038209f5 x1846780895161216/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233680 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233655.729744:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000038209f5 x1846780895161216/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233680 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233655.729749:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233655.729750:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233655.729752:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895161216:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233655.729755:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895161280:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233655.729765:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233655.729768:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233655/real 1761233655] req@00000000d66930c2 x1846780895161280/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233680 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233655.729775:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d66930c2 x1846780895161280/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233680 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233655.729779:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233655.729780:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233655.729782:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895161280:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233655.729785:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895161344:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233655.729797:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233655.729801:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233655/real 1761233655] req@00000000ffe86d04 x1846780895161344/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233680 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233655.729807:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ffe86d04 x1846780895161344/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233680 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233655.729812:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233655.729813:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233655.729815:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895161344:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233655.729818:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895161408:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233655.729827:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233655.729830:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233655/real 1761233655] req@00000000df5b2270 x1846780895161408/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233680 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233655.729837:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df5b2270 x1846780895161408/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233680 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233655.729841:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233655.729842:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233655.729844:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895161408:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233655.729847:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895161472:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233655.729856:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233655.729860:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233655/real 1761233655] req@00000000df0b94a8 x1846780895161472/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233680 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233655.729866:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df0b94a8 x1846780895161472/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233680 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233655.729871:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233655.729872:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233655.729874:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895161472:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233656.497426:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233657.521389:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233658.545407:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233659.569482:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233660.593505:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233660.849405:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233660.849439:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233660.849475:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233660.849489:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6221 00000100:00080000:1.0:1761233660.849494:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233660.849503:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233660.849547:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004361f727 x1846780895161536/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233660.849579:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233660.849583:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233660.849585:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233660.849589:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6221 00000100:00080000:1.0:1761233660.849592:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233660.849596:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233660.849607:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233660.849620:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006a5e9865 x1846780895161600/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233660.849631:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1761233660.849637:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004361f727 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895161536:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233660.849641:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233660.849645:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233660.849654:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6221 00000100:00080000:1.0:1761233660.849657:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233660.849660:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233660.849666:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233660.849681:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac65ecb7 x1846780895161664/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233660.849690:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233660.849694:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233660.849696:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233660.849699:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6221 00000100:00080000:1.0:1761233660.849701:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233660.849704:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233660.849709:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233660.849720:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df0b94a8 x1846780895161728/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233660.849729:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233660.849733:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233660.849734:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233660.849737:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6221 00000100:00080000:1.0:1761233660.849739:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233660.849743:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233660.849748:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1761233660.849755:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233660.849765:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df5b2270 x1846780895161792/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233660.849785:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233660.849790:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233660.849792:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233660.849796:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6221 00000100:00080000:1.0:1761233660.849799:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233660.849803:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233660.849807:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233660.849814:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233660/real 1761233660] req@000000004361f727 x1846780895161536/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233685 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233660.849820:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ffe86d04 x1846780895161856/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233660.849852:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233660.849860:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233660.849863:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233660.849870:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6221 00000100:00080000:1.0:1761233660.849874:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233660.849877:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233660.849882:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233660.849909:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004361f727 x1846780895161536/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233685 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233660.849914:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d66930c2 x1846780895161920/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233660.849960:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233660.849967:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233660.849978:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004361f727 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895161536:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761233660.850023:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a5e9865 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895161600:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761233660.850087:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233660.850105:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233660/real 1761233660] req@000000006a5e9865 x1846780895161600/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233685 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233660.850128:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006a5e9865 x1846780895161600/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233685 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233660.850142:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233660.850146:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233660.850152:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a5e9865 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895161600:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233660.850164:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac65ecb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895161664:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233660.850199:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233660.850212:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233660/real 1761233660] req@00000000ac65ecb7 x1846780895161664/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233685 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233660.850270:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac65ecb7 x1846780895161664/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233685 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233660.850287:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233660.850290:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233660.850294:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac65ecb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895161664:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233660.850301:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895161728:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233660.850330:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233660.850340:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233660/real 1761233660] req@00000000df0b94a8 x1846780895161728/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233685 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233660.850351:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df0b94a8 x1846780895161728/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233685 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233660.850359:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233660.850362:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233660.850365:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895161728:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233660.850370:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895161792:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761233660.850387:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233660.850394:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233660/real 1761233660] req@00000000df5b2270 x1846780895161792/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233685 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233660.850406:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df5b2270 x1846780895161792/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233685 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233660.850414:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233660.850416:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233660.850448:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895161792:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233660.850455:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895161856:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233660.850480:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233660.850488:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233660/real 1761233660] req@00000000ffe86d04 x1846780895161856/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233685 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233660.850499:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ffe86d04 x1846780895161856/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233685 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233660.850507:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233660.850510:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233660.850513:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895161856:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233660.850518:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895161920:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233660.850551:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233660.850557:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233660/real 1761233660] req@00000000d66930c2 x1846780895161920/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233685 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233660.850566:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d66930c2 x1846780895161920/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233685 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233660.850572:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233660.850575:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233660.850577:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895161920:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233661.617373:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233662.641446:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233663.665416:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233664.689395:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233665.713407:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233665.969316:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233665.969329:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233665.969333:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233665.969357:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6226 00000100:00080000:1.0:1761233665.969362:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233665.969370:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233665.969404:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000038209f5 x1846780895161984/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233665.969425:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233665.969427:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233665.969429:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233665.969431:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6226 00000100:00080000:1.0:1761233665.969433:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233665.969436:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233665.969443:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233665.969451:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000880f16b4 x1846780895162048/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761233665.969455:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895161984:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:1.0:1761233665.969460:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233665.969464:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233665.969465:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233665.969468:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6226 00000100:00080000:1.0:1761233665.969470:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233665.969474:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233665.969479:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233665.969487:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cb5be845 x1846780895162112/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233665.969493:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233665.969495:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233665.969496:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233665.969499:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6226 00000100:00080000:1.0:1761233665.969500:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233665.969504:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233665.969506:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233665.969512:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a95774fb x1846780895162176/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233665.969516:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233665.969518:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233665.969519:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233665.969521:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6226 00000100:00080000:1.0:1761233665.969523:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1761233665.969526:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:1.0:1761233665.969528:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233665.969532:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233665.969538:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000c736bb2 x1846780895162240/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233665.969543:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233665.969545:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233665.969546:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233665.969549:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6226 00000100:00080000:1.0:1761233665.969551:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233665.969555:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:2.0:1761233665.969557:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233665/real 1761233665] req@00000000038209f5 x1846780895161984/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233690 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000008:00100000:1.0:1761233665.969558:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233665.969567:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090ac513e x1846780895162304/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233665.969581:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233665.969583:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233665.969584:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233665.969586:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6226 00000100:00080000:1.0:1761233665.969588:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233665.969592:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:2.0:1761233665.969593:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000038209f5 x1846780895161984/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233690 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000008:00100000:1.0:1761233665.969595:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233665.969604:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000053e52331 x1846780895162368/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233665.969610:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233665.969615:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233665.969622:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895161984:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761233665.969645:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895162048:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233665.969664:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233665.969680:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233665/real 1761233665] req@00000000880f16b4 x1846780895162048/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233690 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233665.969690:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000880f16b4 x1846780895162048/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233690 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233665.969696:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233665.969697:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233665.969699:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895162048:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233665.969702:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895162112:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233665.969717:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233665.969721:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233665/real 1761233665] req@00000000cb5be845 x1846780895162112/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233690 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233665.969727:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cb5be845 x1846780895162112/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233690 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233665.969732:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233665.969733:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233665.969735:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895162112:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233665.969738:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895162176:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233665.969747:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233665.969751:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233665/real 1761233665] req@00000000a95774fb x1846780895162176/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233690 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233665.969756:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a95774fb x1846780895162176/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233690 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233665.969761:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233665.969762:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233665.969763:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895162176:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233665.969766:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895162240:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233665.969775:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233665.969779:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233665/real 1761233665] req@000000000c736bb2 x1846780895162240/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233690 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233665.969784:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000000c736bb2 x1846780895162240/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233690 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233665.969789:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233665.969791:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233665.969792:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895162240:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233665.969795:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895162304:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233665.969804:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233665.969807:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233665/real 1761233665] req@0000000090ac513e x1846780895162304/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233690 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233665.969813:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090ac513e x1846780895162304/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233690 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233665.969816:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233665.969818:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233665.969820:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895162304:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233665.969823:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895162368:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233665.969832:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233665.969835:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233665/real 1761233665] req@0000000053e52331 x1846780895162368/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233690 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233665.969841:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000053e52331 x1846780895162368/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233690 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233665.969845:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233665.969846:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233665.969848:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895162368:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233666.737406:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233667.761420:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233668.785426:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233669.809403:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233670.833436:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233671.089340:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233671.089349:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233671.089352:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233671.089360:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6232 00000100:00080000:2.0:1761233671.089363:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233671.089369:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233671.089404:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895162432/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233671.089434:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233671.089438:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233671.089440:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233671.089444:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6232 00000100:00080000:2.0:1761233671.089446:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233671.089450:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233671.089460:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233671.089475:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895162496/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233671.089479:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895162432:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761233671.089493:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233671.089497:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233671.089498:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233671.089503:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6232 00000100:00080000:2.0:1761233671.089506:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233671.089509:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233671.089514:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233671.089526:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895162560/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233671.089535:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233671.089539:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233671.089541:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233671.089544:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6232 00000100:00080000:2.0:1761233671.089546:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233671.089553:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:1.0:1761233671.089554:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:2.0:1761233671.089560:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233671.089572:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895162624/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233671.089582:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233671/real 1761233671] req@00000000ac7d8d5b x1846780895162432/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233696 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233671.089582:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233671.089588:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233671.089590:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233671.089596:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6232 00000100:00080000:2.0:1761233671.089600:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233671.089604:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233671.089608:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233671.089616:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895162432/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233696 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233671.089616:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895162688/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233671.089634:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233671.089639:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233671.089640:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233671.089643:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233671.089644:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6232 00000100:00080000:2.0:1761233671.089647:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233671.089650:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761233671.089652:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233671.089656:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233671.089658:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895162432:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761233671.089664:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895162752/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233671.089670:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233671.089672:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233671.089673:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233671.089676:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6232 00000100:00080000:2.0:1761233671.089678:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233671.089679:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233671.089681:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233671.089688:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895162816/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233671.089692:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895162496:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233671.089723:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233671.089733:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233671/real 1761233671] req@000000004ae5bc9b x1846780895162496/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233696 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233671.089746:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895162496/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233696 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233671.089753:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233671.089756:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233671.089759:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895162496:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233671.089765:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895162560:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233671.089784:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233671.089790:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233671/real 1761233671] req@000000008ebccab5 x1846780895162560/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233696 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233671.089797:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895162560/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233696 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233671.089801:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233671.089803:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233671.089805:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895162560:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233671.089808:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895162624:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233671.089818:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233671.089822:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233671/real 1761233671] req@00000000a1e8c6bb x1846780895162624/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233696 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233671.089828:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895162624/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233696 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233671.089833:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233671.089834:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233671.089836:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895162624:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233671.089839:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895162688:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233671.089848:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233671.089852:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233671/real 1761233671] req@00000000c10cebe6 x1846780895162688/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233696 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233671.089857:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895162688/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233696 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233671.089862:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233671.089863:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233671.089865:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895162688:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233671.089868:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895162752:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233671.089876:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233671.089880:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233671/real 1761233671] req@0000000099b7e5d9 x1846780895162752/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233696 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233671.089885:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895162752/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233696 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233671.089889:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233671.089890:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233671.089892:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895162752:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233671.089907:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895162816:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233671.089919:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233671.089923:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233671/real 1761233671] req@00000000f4e415d3 x1846780895162816/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233696 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233671.089929:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895162816/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233696 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233671.089933:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233671.089934:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233671.089936:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895162816:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233671.857369:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233672.881384:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233673.905365:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233674.929405:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233675.953358:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233676.209298:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233676.209311:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233676.209316:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233676.209328:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6237 00000100:00080000:2.0:1761233676.209332:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233676.209340:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233676.209379:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895162880/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233676.209408:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233676.209411:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233676.209414:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233676.209417:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6237 00000100:00080000:2.0:1761233676.209419:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233676.209423:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233676.209433:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233676.209446:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895162944/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233676.209453:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233676.209456:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233676.209457:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233676.209460:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6237 00000100:00080000:2.0:1761233676.209461:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233676.209463:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233676.209466:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233676.209470:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895162880:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761233676.209475:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895163008/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233676.209483:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233676.209485:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233676.209486:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233676.209488:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6237 00000100:00080000:2.0:1761233676.209489:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233676.209491:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233676.209494:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233676.209501:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895163072/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233676.209506:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233676.209509:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233676.209510:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233676.209513:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6237 00000100:00080000:2.0:1761233676.209514:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233676.209516:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233676.209519:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233676.209524:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895163136/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233676.209529:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233676.209531:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233676.209532:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233676.209534:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6237 00000100:00080000:2.0:1761233676.209535:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233676.209537:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233676.209540:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233676.209546:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895163200/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233676.209550:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233676.209553:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233676.209554:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761233676.209557:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895162944:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:2.0:1761233676.209561:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6237 00000100:00080000:2.0:1761233676.209564:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233676.209567:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233676.209571:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233676.209580:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895163264/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233676.209593:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895163008:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233676.209609:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895163072:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233676.209620:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895163136:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233676.209632:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895163200:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233676.209652:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895163264:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233676.977386:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233678.001370:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233679.025411:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1761233679.281544:0:2209:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:2.0:1761233679.281590:0:2209:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1761233679.281602:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761233679.281644:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761233679.281654:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761233679.281661:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761233679.281670:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761233679.281681:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761233679.281697:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:1.0:1761233679.281701:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233676/real 1761233679] req@00000000fd4aa3a2 x1846780895163008/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233701 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233679.281729:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895163008/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233701 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233679.281741:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233679.281745:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233679.281749:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895163008:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233679.281756:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233676/real 1761233679] req@000000006c1041ba x1846780895163072/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233701 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233679.281766:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895163072/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233701 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233679.281773:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233679.281775:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233679.281778:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895163072:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233679.281783:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233676/real 1761233679] req@000000006d8c3eb0 x1846780895163136/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233701 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233679.281794:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895163136/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233701 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233679.281801:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233679.281804:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233679.281807:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895163136:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233679.281812:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233676/real 1761233679] req@0000000088939481 x1846780895163200/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233701 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233679.281821:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895163200/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233701 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233679.281834:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233679.281837:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233679.281840:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895163200:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233679.281845:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233676/real 1761233679] req@000000006ff1fdbc x1846780895163264/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233701 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233679.281855:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895163264/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233701 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233679.281861:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233679.281863:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233679.281865:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895163264:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233679.281871:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233676/real 1761233679] req@00000000135e9a6e x1846780895162880/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233701 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233679.281881:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895162880/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233701 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233679.281888:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233679.281891:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233679.281893:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895162880:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761233679.281898:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233676/real 1761233679] req@00000000fd3935a6 x1846780895162944/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233701 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233679.281907:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895162944/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233701 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233679.281918:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233679.281920:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233679.281923:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895162944:192.168.201.157@tcp:38:kworker/u8:2.0 00010000:00010000:1.0:1761233680.049419:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233681.073380:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233681.329283:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233681.329295:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233681.329299:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233681.329309:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6242 00000100:00080000:2.0:1761233681.329313:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233681.329318:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233681.329347:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895163328/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233681.329370:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233681.329373:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233681.329374:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233681.329376:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6242 00000100:00080000:2.0:1761233681.329378:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233681.329381:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233681.329388:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233681.329395:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895163392/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233681.329400:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233681.329402:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233681.329403:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233681.329405:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6242 00000100:00080000:2.0:1761233681.329406:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233681.329408:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233681.329410:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233681.329417:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895163456/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233681.329422:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233681.329424:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233681.329425:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233681.329428:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6242 00000100:00080000:2.0:1761233681.329429:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233681.329431:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233681.329434:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233681.329441:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895163520/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233681.329446:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895163328:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761233681.329448:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233681.329453:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233681.329454:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233681.329458:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6242 00000100:00080000:2.0:1761233681.329459:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233681.329461:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233681.329464:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233681.329472:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895163584/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233681.329477:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233681.329479:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233681.329481:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233681.329483:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6242 00000100:00080000:2.0:1761233681.329485:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233681.329487:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233681.329490:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233681.329496:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895163648/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233681.329501:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233681.329502:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233681.329503:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233681.329505:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6242 00000100:00080000:2.0:1761233681.329506:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233681.329511:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:1.0:1761233681.329513:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:2.0:1761233681.329516:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233681.329526:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895163712/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233681.329535:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233681/real 1761233681] req@000000004f29ca26 x1846780895163328/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233706 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233681.329552:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895163328/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233706 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233681.329562:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233681.329565:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233681.329569:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895163328:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233681.329576:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895163392:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233681.329601:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233681.329610:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233681/real 1761233681] req@0000000033765045 x1846780895163392/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233706 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233681.329620:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895163392/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233706 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233681.329627:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233681.329630:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233681.329633:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895163392:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233681.329638:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895163456:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233681.329659:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233681.329667:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233681/real 1761233681] req@000000009d315fa5 x1846780895163456/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233706 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233681.329678:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895163456/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233706 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233681.329685:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233681.329687:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233681.329690:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895163456:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233681.329720:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895163520:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233681.329739:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233681.329747:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233681/real 1761233681] req@0000000040d5f361 x1846780895163520/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233706 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233681.329765:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895163520/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233706 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233681.329772:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233681.329774:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233681.329777:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895163520:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233681.329783:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895163584:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233681.329802:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233681.329809:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233681/real 1761233681] req@000000007dcf84ae x1846780895163584/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233706 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233681.329820:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895163584/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233706 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233681.329827:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233681.329830:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233681.329833:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895163584:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233681.329838:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895163648:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233681.329857:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233681.329865:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233681/real 1761233681] req@000000007091f547 x1846780895163648/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233706 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233681.329889:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895163648/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233706 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233681.329896:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233681.329899:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233681.329902:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895163648:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233681.329915:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895163712:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233681.329938:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233681.329946:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233681/real 1761233681] req@0000000064a2cea0 x1846780895163712/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233706 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233681.329956:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895163712/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233706 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233681.329962:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233681.329965:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233681.329967:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895163712:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233682.097410:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233683.121419:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233684.145448:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233685.169381:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233686.193447:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233686.449398:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233686.449410:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233686.449414:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233686.449426:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6247 00000100:00080000:0.0:1761233686.449431:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233686.449438:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233686.449487:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000addc3db8 x1846780895163776/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233686.449511:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233686.449514:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233686.449516:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233686.449520:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6247 00000100:00080000:0.0:1761233686.449522:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233686.449524:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233686.449534:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233686.449543:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000078e7052d x1846780895163840/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233686.449551:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233686.449554:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233686.449556:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233686.449558:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6247 00000100:00080000:0.0:1761233686.449559:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233686.449561:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233686.449565:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233686.449573:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895163904/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233686.449578:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895163776:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233686.449581:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233686.449586:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233686.449588:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233686.449591:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6247 00000100:00080000:0.0:1761233686.449592:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233686.449594:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233686.449596:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233686.449603:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895163968/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233686.449608:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233686.449610:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233686.449611:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233686.449613:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6247 00000100:00080000:0.0:1761233686.449614:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233686.449617:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233686.449619:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233686.449626:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895164032/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233686.449631:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233686.449633:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233686.449634:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233686.449636:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6247 00000100:00080000:0.0:1761233686.449637:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233686.449639:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233686.449641:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233686.449647:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895164096/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233686.449653:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233686.449655:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233686.449656:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233686.449658:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6247 00000100:00080000:0.0:1761233686.449659:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233686.449660:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233686.449663:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761233686.449666:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233686.449673:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895164160/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233686.449692:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233686/real 1761233686] req@00000000addc3db8 x1846780895163776/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233711 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233686.449707:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000addc3db8 x1846780895163776/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233711 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233686.449716:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233686.449719:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233686.449723:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895163776:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233686.449729:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895163840:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233686.449753:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233686.449761:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233686/real 1761233686] req@0000000078e7052d x1846780895163840/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233711 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233686.449773:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000078e7052d x1846780895163840/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233711 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233686.449781:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233686.449783:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233686.449787:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895163840:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233686.449814:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895163904:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233686.449835:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233686.449843:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233686/real 1761233686] req@000000005beaae31 x1846780895163904/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233711 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233686.449855:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895163904/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233711 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233686.449863:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233686.449865:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233686.449868:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895163904:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233686.449872:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895163968:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233686.449890:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233686.449898:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233686/real 1761233686] req@0000000034d49f59 x1846780895163968/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233711 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233686.449909:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895163968/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233711 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233686.449917:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233686.449919:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233686.449922:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895163968:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233686.449928:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895164032:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233686.449948:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233686.449955:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233686/real 1761233686] req@00000000eca9ddc2 x1846780895164032/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233711 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233686.449966:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895164032/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233711 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233686.449975:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233686.449978:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233686.449981:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895164032:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233686.449986:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895164096:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233686.450000:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233686.450006:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233686/real 1761233686] req@000000006f2efc8b x1846780895164096/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233711 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233686.450046:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895164096/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233711 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233686.450052:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233686.450054:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233686.450056:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895164096:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233686.450060:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895164160:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233686.450075:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233686.450080:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233686/real 1761233686] req@000000004862b808 x1846780895164160/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233711 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233686.450087:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895164160/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233711 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233686.450092:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233686.450094:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233686.450096:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895164160:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233687.217350:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233688.241451:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233689.265382:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233690.289456:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233691.313354:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233691.569318:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233691.569327:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233691.569332:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233691.569341:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6252 00000100:00080000:0.0:1761233691.569345:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233691.569350:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233691.569382:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895164224/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233691.569401:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233691.569405:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233691.569406:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233691.569409:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6252 00000100:00080000:0.0:1761233691.569411:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233691.569415:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233691.569424:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233691.569433:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895164288/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233691.569439:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233691.569443:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1761233691.569443:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895164224:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233691.569445:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233691.569452:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6252 00000100:00080000:0.0:1761233691.569454:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233691.569456:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233691.569459:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233691.569466:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895164352/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233691.569471:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233691.569473:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233691.569474:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233691.569477:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6252 00000100:00080000:0.0:1761233691.569478:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233691.569480:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233691.569483:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233691.569489:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895164416/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233691.569493:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233691.569496:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233691.569496:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233691.569499:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6252 00000100:00080000:0.0:1761233691.569500:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233691.569503:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233691.569505:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233691.569510:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895164480/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233691.569515:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233691.569517:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233691.569518:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1761233691.569523:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233691.569524:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6252 00000100:00080000:0.0:1761233691.569527:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233691.569529:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233691.569532:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233691.569540:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895164544/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233691.569545:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233691.569547:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233691.569548:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1761233691.569552:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233691/real 1761233691] req@00000000c6e06344 x1846780895164224/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233716 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233691.569553:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6252 00000100:00080000:0.0:1761233691.569556:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233691.569561:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233691.569565:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233691.569573:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895164608/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761233691.569582:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895164224/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233716 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233691.569596:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233691.569599:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233691.569604:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895164224:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761233691.569628:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895164288:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761233691.569654:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233691.569663:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233691/real 1761233691] req@000000005593c994 x1846780895164288/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233716 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233691.569674:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895164288/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233716 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233691.569681:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233691.569683:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233691.569686:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895164288:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233691.569692:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895164352:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233691.569712:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233691.569721:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233691/real 1761233691] req@000000001f9ec359 x1846780895164352/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233716 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233691.569732:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895164352/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233716 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233691.569740:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233691.569742:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233691.569745:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895164352:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233691.569750:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895164416:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233691.569768:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233691.569775:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233691/real 1761233691] req@00000000efa29eb9 x1846780895164416/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233716 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233691.569786:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895164416/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233716 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233691.569793:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233691.569795:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233691.569798:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895164416:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233691.569803:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895164480:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761233691.569823:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233691.569831:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233691/real 1761233691] req@000000006d61dd06 x1846780895164480/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233716 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233691.569841:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895164480/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233716 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233691.569856:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233691.569859:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233691.569861:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895164480:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233691.569867:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895164544:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233691.569886:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233691.569894:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233691/real 1761233691] req@00000000ef655201 x1846780895164544/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233716 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233691.569905:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895164544/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233716 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233691.569912:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233691.569914:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233691.569917:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895164544:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233691.569922:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895164608:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233691.569978:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233691.569987:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233691/real 1761233691] req@000000004130b6bd x1846780895164608/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233716 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233691.569998:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895164608/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233716 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233691.570006:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233691.570008:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233691.570011:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895164608:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233692.337386:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233693.361325:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233694.385463:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233695.409424:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233696.433449:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233696.689344:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233696.689353:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233696.689356:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233696.689367:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6257 00000100:00080000:1.0:1761233696.689370:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233696.689375:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233696.689407:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000568f2a96 x1846780895164672/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233696.689427:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233696.689430:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233696.689431:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233696.689433:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6257 00000100:00080000:1.0:1761233696.689435:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233696.689437:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233696.689444:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233696.689452:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c652d0ad x1846780895164736/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233696.689459:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1761233696.689462:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895164672:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:1.0:1761233696.689463:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233696.689466:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233696.689473:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6257 00000100:00080000:1.0:1761233696.689474:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233696.689477:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233696.689480:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233696.689488:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ba3340f x1846780895164800/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233696.689493:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233696.689496:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233696.689497:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233696.689499:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6257 00000100:00080000:1.0:1761233696.689500:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233696.689503:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233696.689506:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233696.689512:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d74b2e3c x1846780895164864/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233696.689517:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233696.689519:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233696.689520:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233696.689523:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6257 00000100:00080000:1.0:1761233696.689524:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233696.689526:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233696.689529:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1761233696.689531:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233696.689539:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001aedf9b1 x1846780895164928/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233696.689546:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233696.689548:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233696.689549:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233696.689562:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6257 00000100:00080000:1.0:1761233696.689565:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1761233696.689565:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233696/real 1761233696] req@00000000568f2a96 x1846780895164672/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233721 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233696.689572:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233696.689577:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233696.689587:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a611ee34 x1846780895164992/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761233696.689590:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000568f2a96 x1846780895164672/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233721 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233696.689609:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233696.689613:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233696.689613:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233696.689615:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233696.689628:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6257 00000100:00080000:2.0:1761233696.689629:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:1.0:1761233696.689633:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1761233696.689635:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895164672:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:1.0:1761233696.689640:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233696.689644:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233696.689651:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000328368c8 x1846780895165056/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761233696.689664:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895164736:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233696.689683:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233696.689689:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233696/real 1761233696] req@00000000c652d0ad x1846780895164736/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233721 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233696.689697:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c652d0ad x1846780895164736/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233721 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233696.689701:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233696.689703:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233696.689705:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895164736:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233696.689708:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895164800:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233696.689719:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233696.689723:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233696/real 1761233696] req@000000007ba3340f x1846780895164800/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233721 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233696.689734:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007ba3340f x1846780895164800/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233721 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233696.689738:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233696.689740:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233696.689741:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895164800:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233696.689744:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895164864:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233696.689753:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233696.689757:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233696/real 1761233696] req@00000000d74b2e3c x1846780895164864/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233721 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233696.689763:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d74b2e3c x1846780895164864/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233721 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233696.689767:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233696.689769:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233696.689770:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895164864:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233696.689774:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895164928:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233696.689784:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233696.689788:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233696/real 1761233696] req@000000001aedf9b1 x1846780895164928/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233721 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233696.689794:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001aedf9b1 x1846780895164928/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233721 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233696.689799:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233696.689801:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233696.689802:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895164928:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233696.689806:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895164992:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233696.689815:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233696.689819:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233696/real 1761233696] req@00000000a611ee34 x1846780895164992/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233721 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233696.689825:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a611ee34 x1846780895164992/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233721 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233696.689829:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233696.689831:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233696.689832:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895164992:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233696.689835:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895165056:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233696.689844:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233696.689847:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233696/real 1761233696] req@00000000328368c8 x1846780895165056/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233721 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233696.689853:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000328368c8 x1846780895165056/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233721 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233696.689880:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233696.689882:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233696.689884:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895165056:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233697.457371:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233698.481430:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233699.505396:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233700.529416:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233701.553419:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233701.809411:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233701.809424:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233701.809429:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233701.809441:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6262 00000100:00080000:0.0:1761233701.809445:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233701.809452:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233701.809497:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895165120/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233701.809528:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233701.809531:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233701.809532:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233701.809534:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6262 00000100:00080000:0.0:1761233701.809536:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233701.809539:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233701.809549:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233701.809561:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895165184/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233701.809568:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233701.809571:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233701.809572:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233701.809577:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6262 00000100:00080000:0.0:1761233701.809578:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233701.809582:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:2.0:1761233701.809589:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895165120:192.168.201.157@tcp:250:kworker/u8:3.0 00000008:00100000:0.0:1761233701.809592:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233701.809606:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895165248/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233701.809614:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233701.809617:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233701.809617:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233701.809620:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6262 00000100:00080000:0.0:1761233701.809621:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233701.809623:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233701.809625:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233701.809634:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895165312/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233701.809638:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233701.809640:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233701.809641:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233701.809644:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6262 00000100:00080000:0.0:1761233701.809645:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233701.809648:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233701.809651:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233701.809658:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895165376/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233701.809662:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233701.809665:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233701.809666:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233701.809670:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6262 00000100:00080000:0.0:1761233701.809673:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233701.809676:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233701.809678:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233701.809685:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895165440/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:2.0:1761233701.809695:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233701.809702:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233701.809705:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233701.809706:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233701.809710:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6262 00000100:00080000:0.0:1761233701.809712:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233701.809716:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233701.809719:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233701.809727:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895165504/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761233701.809733:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233701/real 1761233701] req@000000009d1c7f73 x1846780895165120/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233726 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233701.809801:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895165120/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233726 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233701.809815:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233701.809819:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233701.809824:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895165120:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761233701.809853:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895165184:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761233701.809881:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233701.809890:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233701/real 1761233701] req@00000000d67bb589 x1846780895165184/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233726 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233701.809907:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895165184/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233726 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233701.809916:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233701.809920:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233701.809923:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895165184:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233701.809930:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895165248:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233701.809952:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233701.809961:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233701/real 1761233701] req@00000000a4a2c44a x1846780895165248/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233726 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233701.809973:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895165248/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233726 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233701.809981:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233701.809984:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233701.809987:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895165248:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233701.809993:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895165312:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233701.810015:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233701.810023:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233701/real 1761233701] req@000000004a74da19 x1846780895165312/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233726 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233701.810035:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895165312/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233726 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233701.810042:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233701.810044:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233701.810047:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895165312:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233701.810053:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895165376:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761233701.810074:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233701.810081:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233701/real 1761233701] req@000000002d1b8ef7 x1846780895165376/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233726 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233701.810092:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895165376/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233726 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233701.810100:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233701.810102:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233701.810105:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895165376:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761233701.810111:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895165440:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233701.810125:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233701.810130:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233701/real 1761233701] req@000000007bd1ec64 x1846780895165440/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233726 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233701.810136:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895165440/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233726 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233701.810141:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233701.810143:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233701.810144:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895165440:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761233701.810147:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895165504:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761233701.810157:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233701.810160:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233701/real 1761233701] req@000000001db96463 x1846780895165504/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233726 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233701.810166:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895165504/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233726 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233701.810171:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233701.810173:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233701.810176:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895165504:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233702.577372:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233703.601432:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233704.625418:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233705.649439:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233706.673390:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233706.929355:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233706.929366:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233706.929371:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233706.929385:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6267 00000100:00080000:0.0:1761233706.929391:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233706.929400:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233706.929455:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895165568/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233706.929485:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233706.929491:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233706.929493:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233706.929498:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6267 00000100:00080000:0.0:1761233706.929500:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233706.929505:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233706.929519:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233706.929536:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895165632/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761233706.929545:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895165568:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233706.929562:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233706.929568:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233706.929570:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233706.929578:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6267 00000100:00080000:0.0:1761233706.929581:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233706.929585:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233706.929590:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233706.929607:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895165696/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233706.929615:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233706.929619:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233706.929620:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233706.929623:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6267 00000100:00080000:0.0:1761233706.929624:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1761233706.929633:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233706.929634:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233706.929645:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233706.929658:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895165760/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233706.929664:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1761233706.929666:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233706/real 1761233706] req@000000005d83493f x1846780895165568/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233731 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233706.929669:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233706.929672:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233706.929684:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6267 00000100:00080000:0.0:1761233706.929692:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233706.929712:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233706.929717:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233706.929726:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005d83493f x1846780895165568/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233731 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233706.929731:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895165824/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233706.929794:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233706.929795:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233706.929806:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233706.929806:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233706.929812:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1761233706.929818:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895165568:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233706.929825:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6267 00000100:00080000:0.0:1761233706.929834:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233706.929837:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233706.929841:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233706.929851:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895165888/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233706.929859:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233706.929861:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1761233706.929862:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895165632:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:0.0:1761233706.929864:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233706.929875:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6267 00000100:00080000:0.0:1761233706.929878:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233706.929882:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233706.929887:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1761233706.929897:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233706.929900:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895165952/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761233706.929910:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233706/real 1761233706] req@00000000f3ed2588 x1846780895165632/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233731 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233706.929922:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f3ed2588 x1846780895165632/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233731 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233706.929929:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233706.929932:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233706.929935:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895165632:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233706.929940:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895165696:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233706.929956:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233706.929961:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233706/real 1761233706] req@000000001db96463 x1846780895165696/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233731 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233706.929969:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895165696/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233731 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233706.929975:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233706.929977:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233706.929979:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895165696:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233706.929984:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895165760:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233706.929996:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233706.930001:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233706/real 1761233706] req@000000007bd1ec64 x1846780895165760/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233731 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233706.930008:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895165760/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233731 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233706.930014:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233706.930016:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233706.930018:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895165760:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233706.930022:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895165824:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233706.930035:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233706.930040:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233706/real 1761233706] req@000000002d1b8ef7 x1846780895165824/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233731 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233706.930047:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895165824/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233731 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233706.930053:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233706.930055:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233706.930057:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895165824:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233706.930073:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895165888:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233706.930088:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233706.930094:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233706/real 1761233706] req@000000004a74da19 x1846780895165888/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233731 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233706.930102:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895165888/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233731 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233706.930107:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233706.930109:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233706.930111:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895165888:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233706.930115:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895165952:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233706.930127:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233706.930132:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233706/real 1761233706] req@00000000a4a2c44a x1846780895165952/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233731 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233706.930140:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895165952/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233731 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233706.930146:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233706.930148:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233706.930149:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895165952:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233707.697410:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233708.721449:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233709.745383:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233710.769362:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233711.793430:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233712.049369:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233712.049384:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233712.049389:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233712.049403:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6272 00000100:00080000:1.0:1761233712.049408:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233712.049416:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233712.049467:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090800dde x1846780895166016/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233712.049501:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233712.049506:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233712.049508:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233712.049512:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6272 00000100:00080000:1.0:1761233712.049514:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233712.049517:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233712.049529:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233712.049541:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e467e107 x1846780895166080/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233712.049550:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233712.049553:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233712.049555:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233712.049559:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6272 00000100:00080000:1.0:1761233712.049560:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233712.049565:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233712.049569:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233712.049579:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cbad49de x1846780895166144/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233712.049588:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233712.049593:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233712.049594:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1761233712.049598:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090800dde pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895166016:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:1.0:1761233712.049602:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6272 00000100:00080000:1.0:1761233712.049606:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233712.049610:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233712.049629:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233712.049642:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000328368c8 x1846780895166208/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233712.049651:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233712.049655:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233712.049657:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233712.049660:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6272 00000100:00080000:1.0:1761233712.049663:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233712.049667:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233712.049672:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233712.049682:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a611ee34 x1846780895166272/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233712.049689:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233712.049693:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233712.049695:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233712.049698:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6272 00000100:00080000:1.0:1761233712.049701:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233712.049707:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:2.0:1761233712.049708:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:1.0:1761233712.049713:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233712.049725:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001aedf9b1 x1846780895166336/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233712.049734:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1761233712.049738:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233712/real 1761233712] req@0000000090800dde x1846780895166016/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233737 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233712.049741:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233712.049743:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233712.049751:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6272 00000100:00080000:1.0:1761233712.049755:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233712.049760:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:2.0:1761233712.049760:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090800dde x1846780895166016/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233737 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000008:00100000:1.0:1761233712.049767:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1761233712.049776:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233712.049780:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233712.049783:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d74b2e3c x1846780895166400/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761233712.049786:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090800dde pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895166016:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761233712.049797:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e467e107 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895166080:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233712.049817:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233712.049823:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233712/real 1761233712] req@00000000e467e107 x1846780895166080/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233737 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233712.049830:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e467e107 x1846780895166080/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233737 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233712.049835:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233712.049837:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233712.049839:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e467e107 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895166080:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233712.049863:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbad49de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895166144:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233712.049873:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233712.049877:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233712/real 1761233712] req@00000000cbad49de x1846780895166144/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233737 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233712.049884:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cbad49de x1846780895166144/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233737 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233712.049888:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233712.049889:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233712.049891:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbad49de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895166144:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233712.049895:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895166208:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233712.049905:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233712.049908:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233712/real 1761233712] req@00000000328368c8 x1846780895166208/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233737 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233712.049915:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000328368c8 x1846780895166208/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233737 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233712.049919:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233712.049920:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233712.049922:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895166208:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233712.049925:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895166272:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761233712.049934:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233712.049938:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233712/real 1761233712] req@00000000a611ee34 x1846780895166272/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233737 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233712.049943:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a611ee34 x1846780895166272/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233737 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233712.049947:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233712.049949:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233712.049951:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895166272:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233712.049954:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895166336:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233712.049963:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233712.049966:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233712/real 1761233712] req@000000001aedf9b1 x1846780895166336/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233737 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233712.049973:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001aedf9b1 x1846780895166336/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233737 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233712.049978:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233712.049979:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233712.049980:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895166336:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233712.049983:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895166400:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761233712.049992:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233712.049995:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233712/real 1761233712] req@00000000d74b2e3c x1846780895166400/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233737 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233712.050001:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d74b2e3c x1846780895166400/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233737 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233712.050004:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233712.050006:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233712.050007:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895166400:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233712.817404:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233713.841440:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233714.865407:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233715.889468:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233716.913452:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233717.169347:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233717.169359:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233717.169364:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233717.169377:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6278 00000100:00080000:2.0:1761233717.169382:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233717.169389:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233717.169440:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895166464/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233717.169474:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233717.169477:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233717.169478:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233717.169481:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6278 00000100:00080000:2.0:1761233717.169483:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233717.169485:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233717.169494:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233717.169503:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895166528/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233717.169509:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233717.169512:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233717.169513:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233717.169516:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6278 00000100:00080000:2.0:1761233717.169517:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233717.169520:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233717.169524:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233717.169530:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895166464:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761233717.169547:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895166592/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233717.169555:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233717.169559:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233717.169560:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233717.169562:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6278 00000100:00080000:2.0:1761233717.169564:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233717.169568:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233717.169572:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233717.169580:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895166656/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233717.169586:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233717.169589:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233717.169590:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233717.169593:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6278 00000100:00080000:2.0:1761233717.169594:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233717.169596:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233717.169599:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233717.169604:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895166720/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233717.169609:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233717.169611:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233717.169612:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233717.169614:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6278 00000100:00080000:2.0:1761233717.169615:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233717.169617:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233717.169621:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761233717.169624:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233717.169630:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895166784/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233717.169636:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233717.169639:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233717.169642:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233717.169645:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6278 00000100:00080000:2.0:1761233717.169647:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233717.169650:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761233717.169653:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233717/real 1761233717] req@0000000064a2cea0 x1846780895166464/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233742 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:2.0:1761233717.169655:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233717.169666:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895166848/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233717.169679:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895166464/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233742 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233717.169689:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233717.169692:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233717.169695:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895166464:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233717.169702:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895166528:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233717.169721:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233717.169726:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233717/real 1761233717] req@000000007091f547 x1846780895166528/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233742 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233717.169738:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895166528/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233742 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233717.169743:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233717.169745:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233717.169747:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895166528:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233717.169781:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895166592:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233717.169803:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233717.169811:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233717/real 1761233717] req@000000007dcf84ae x1846780895166592/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233742 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233717.169820:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895166592/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233742 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233717.169827:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233717.169829:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233717.169832:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895166592:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233717.169838:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895166656:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233717.169854:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233717.169860:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233717/real 1761233717] req@0000000040d5f361 x1846780895166656/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233742 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233717.169870:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895166656/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233742 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233717.169877:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233717.169880:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233717.169883:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895166656:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233717.169888:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895166720:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233717.169905:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233717.169912:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233717/real 1761233717] req@000000009d315fa5 x1846780895166720/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233742 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233717.169923:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895166720/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233742 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233717.169931:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233717.169933:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233717.169935:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895166720:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233717.169940:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895166784:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233717.169954:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233717.169959:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233717/real 1761233717] req@0000000033765045 x1846780895166784/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233742 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233717.169965:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895166784/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233742 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233717.169971:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233717.169972:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233717.169974:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895166784:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233717.169977:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895166848:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233717.169988:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233717.169992:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233717/real 1761233717] req@000000004f29ca26 x1846780895166848/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233742 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233717.169999:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895166848/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233742 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233717.170003:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233717.170004:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233717.170006:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895166848:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233717.937470:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233718.961409:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233719.985426:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233721.009453:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233722.033397:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233722.289327:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233722.289339:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233722.289344:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233722.289355:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6283 00000100:00080000:2.0:1761233722.289359:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233722.289366:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233722.289402:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895166912/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233722.289424:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233722.289426:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233722.289427:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233722.289430:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6283 00000100:00080000:2.0:1761233722.289431:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233722.289434:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233722.289442:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233722.289451:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895166976/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233722.289473:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233722.289476:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233722.289477:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233722.289480:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6283 00000100:00080000:2.0:1761233722.289481:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761233722.289486:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895166912:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761233722.289486:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233722.289493:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233722.289502:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895167040/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233722.289508:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233722.289511:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233722.289512:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233722.289514:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6283 00000100:00080000:2.0:1761233722.289516:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233722.289518:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233722.289520:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233722.289526:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895167104/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233722.289531:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233722.289532:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233722.289533:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233722.289535:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6283 00000100:00080000:2.0:1761233722.289536:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233722.289539:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233722.289542:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761233722.289549:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233722.289553:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895167168/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233722.289563:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233722.289566:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233722.289568:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761233722.289571:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233722/real 1761233722] req@00000000fd3935a6 x1846780895166912/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233747 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233722.289576:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6283 00000100:00080000:2.0:1761233722.289581:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233722.289586:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761233722.289592:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895166912/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233747 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000008:00100000:2.0:1761233722.289592:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1761233722.289603:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233722.289606:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233722.289606:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895167232/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233722.289611:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895166912:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761233722.289621:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233722.289622:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895166976:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:2.0:1761233722.289626:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233722.289628:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233722.289632:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6283 00000100:00080000:2.0:1761233722.289634:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233722.289637:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233722.289640:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761233722.289643:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233722.289651:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895167296/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233722.289653:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233722/real 1761233722] req@00000000135e9a6e x1846780895166976/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233747 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233722.289678:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895166976/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233747 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233722.289684:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233722.289685:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233722.289688:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895166976:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233722.289707:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895167040:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233722.289719:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233722.289723:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233722/real 1761233722] req@000000006ff1fdbc x1846780895167040/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233747 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233722.289729:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895167040/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233747 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233722.289734:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233722.289735:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233722.289736:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895167040:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233722.289739:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895167104:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233722.289749:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233722.289752:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233722/real 1761233722] req@0000000088939481 x1846780895167104/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233747 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233722.289758:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895167104/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233747 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233722.289763:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233722.289764:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233722.289766:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895167104:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233722.289769:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895167168:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233722.289778:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233722.289782:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233722/real 1761233722] req@000000006d8c3eb0 x1846780895167168/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233747 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233722.289788:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895167168/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233747 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233722.289792:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233722.289793:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233722.289795:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895167168:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233722.289798:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895167232:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233722.289806:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233722.289810:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233722/real 1761233722] req@000000006c1041ba x1846780895167232/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233747 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233722.289815:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895167232/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233747 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233722.289819:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233722.289820:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233722.289822:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895167232:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233722.289825:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895167296:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233722.289836:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233722.289839:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233722/real 1761233722] req@00000000fd4aa3a2 x1846780895167296/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233747 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233722.289845:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895167296/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233747 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233722.289851:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233722.289853:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233722.289856:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895167296:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233723.057420:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233724.081375:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233725.105452:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233726.129361:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233727.153410:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233727.409300:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233727.409310:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233727.409314:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233727.409323:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6288 00000100:00080000:2.0:1761233727.409326:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233727.409332:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233727.409364:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895167360/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233727.409414:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233727.409417:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233727.409418:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233727.409420:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6288 00000100:00080000:2.0:1761233727.409422:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233727.409425:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233727.409433:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233727.409442:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895167424/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233727.409448:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233727.409450:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233727.409450:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233727.409452:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6288 00000100:00080000:2.0:1761233727.409453:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233727.409455:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233727.409458:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233727.409465:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895167488/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233727.409470:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233727.409472:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233727.409473:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233727.409476:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6288 00000100:00080000:2.0:1761233727.409477:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233727.409479:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233727.409481:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233727.409487:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895167552/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233727.409492:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895167360:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761233727.409495:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233727.409500:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233727.409501:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233727.409505:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6288 00000100:00080000:2.0:1761233727.409507:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233727.409509:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233727.409511:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233727.409520:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895167616/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233727.409525:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233727.409527:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233727.409528:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233727.409530:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6288 00000100:00080000:2.0:1761233727.409532:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233727.409534:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233727.409537:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233727.409542:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895167680/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233727.409547:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233727.409549:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233727.409550:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233727.409552:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6288 00000100:00080000:2.0:1761233727.409553:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233727.409555:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233727.409558:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233727.409564:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895167744/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233727.409571:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233727.409596:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233727/real 1761233727] req@00000000f4e415d3 x1846780895167360/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233752 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233727.409614:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895167360/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233752 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233727.409624:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233727.409627:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233727.409631:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895167360:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233727.409639:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895167424:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233727.409660:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233727.409668:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233727/real 1761233727] req@0000000099b7e5d9 x1846780895167424/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233752 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233727.409678:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895167424/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233752 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233727.409685:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233727.409687:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233727.409691:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895167424:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233727.409697:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895167488:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233727.409713:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233727.409720:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233727/real 1761233727] req@00000000c10cebe6 x1846780895167488/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233752 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233727.409731:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895167488/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233752 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233727.409737:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233727.409740:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233727.409742:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895167488:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233727.409769:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895167552:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233727.409787:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233727.409802:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233727/real 1761233727] req@00000000a1e8c6bb x1846780895167552/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233752 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233727.409814:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895167552/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233752 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233727.409822:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233727.409824:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233727.409827:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895167552:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233727.409832:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895167616:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233727.409850:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233727.409856:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233727/real 1761233727] req@000000008ebccab5 x1846780895167616/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233752 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233727.409862:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895167616/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233752 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233727.409867:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233727.409868:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233727.409870:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895167616:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233727.409873:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895167680:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233727.409882:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233727.409885:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233727/real 1761233727] req@000000004ae5bc9b x1846780895167680/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233752 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233727.409891:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895167680/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233752 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233727.409896:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233727.409897:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233727.409898:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895167680:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233727.409901:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895167744:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233727.409912:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233727.409918:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233727/real 1761233727] req@00000000ac7d8d5b x1846780895167744/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233752 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233727.409928:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895167744/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233752 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233727.409942:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233727.409944:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233727.409947:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895167744:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233728.177342:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233729.201440:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233730.225376:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233731.249458:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233732.273408:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233732.529316:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233732.529326:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233732.529329:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233732.529339:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6293 00000100:00080000:2.0:1761233732.529342:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233732.529348:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233732.529381:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895167808/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233732.529406:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233732.529409:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233732.529410:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233732.529412:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6293 00000100:00080000:2.0:1761233732.529414:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233732.529416:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233732.529425:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233732.529432:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895167872/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233732.529438:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233732.529441:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233732.529442:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233732.529445:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6293 00000100:00080000:2.0:1761233732.529446:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233732.529448:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233732.529451:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233732.529457:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895167936/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233732.529462:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233732.529465:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1761233732.529466:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895167808:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761233732.529467:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233732.529472:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6293 00000100:00080000:2.0:1761233732.529475:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233732.529477:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233732.529479:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233732.529485:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895168000/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233732.529489:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233732.529491:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233732.529492:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233732.529495:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6293 00000100:00080000:2.0:1761233732.529496:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233732.529499:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233732.529502:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233732.529509:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895168064/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233732.529513:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233732.529516:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233732.529517:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233732.529519:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6293 00000100:00080000:2.0:1761233732.529520:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233732.529522:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233732.529525:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233732.529537:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895168128/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233732.529545:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233732.529547:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233732.529548:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233732.529550:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6293 00000100:00080000:2.0:1761233732.529551:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233732.529557:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:3.0:1761233732.529559:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:2.0:1761233732.529562:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233732.529570:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895168192/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233732.529584:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233732/real 1761233732] req@00000000ac7d8d5b x1846780895167808/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233757 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233732.529599:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895167808/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233757 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233732.529606:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233732.529607:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233732.529610:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895167808:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233732.529615:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895167872:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233732.529631:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233732.529635:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233732/real 1761233732] req@000000004ae5bc9b x1846780895167872/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233757 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233732.529642:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895167872/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233757 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233732.529650:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233732.529652:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233732.529654:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895167872:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233732.529674:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895167936:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233732.529685:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233732.529689:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233732/real 1761233732] req@000000008ebccab5 x1846780895167936/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233757 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233732.529695:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895167936/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233757 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233732.529699:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233732.529701:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233732.529702:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895167936:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233732.529705:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895168000:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233732.529714:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233732.529718:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233732/real 1761233732] req@00000000a1e8c6bb x1846780895168000/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233757 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233732.529724:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895168000/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233757 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233732.529728:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233732.529729:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233732.529731:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895168000:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233732.529734:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895168064:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233732.529743:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233732.529747:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233732/real 1761233732] req@00000000c10cebe6 x1846780895168064/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233757 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233732.529753:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895168064/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233757 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233732.529757:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233732.529758:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233732.529760:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895168064:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233732.529762:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895168128:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233732.529771:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233732.529775:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233732/real 1761233732] req@0000000099b7e5d9 x1846780895168128/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233757 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233732.529780:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895168128/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233757 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233732.529785:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233732.529786:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233732.529788:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895168128:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233732.529790:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895168192:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233732.529799:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233732.529803:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233732/real 1761233732] req@00000000f4e415d3 x1846780895168192/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233757 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233732.529810:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895168192/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233757 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233732.529814:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233732.529815:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233732.529816:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895168192:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233733.297457:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233734.321408:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233735.345364:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233736.369427:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233737.393446:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233737.649328:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233737.649337:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233737.649340:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233737.649349:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6298 00000100:00080000:2.0:1761233737.649352:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233737.649358:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233737.649390:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895168256/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233737.649412:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233737.649416:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233737.649417:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233737.649420:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6298 00000100:00080000:2.0:1761233737.649423:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233737.649426:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233737.649436:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233737.649448:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895168320/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233737.649456:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233737.649459:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233737.649461:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233737.649465:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6298 00000100:00080000:2.0:1761233737.649467:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233737.649472:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233737.649472:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895168256:192.168.201.157@tcp:250:kworker/u8:2.0 00000008:00100000:2.0:1761233737.649479:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233737.649487:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895168384/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233737.649495:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233737.649498:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233737.649499:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233737.649502:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6298 00000100:00080000:2.0:1761233737.649504:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233737.649507:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233737.649510:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233737.649519:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895168448/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233737.649525:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233737.649528:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233737.649529:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233737.649532:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6298 00000100:00080000:2.0:1761233737.649534:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1761233737.649537:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233737.649538:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233737.649542:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233737.649548:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895168512/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233737.649555:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233737.649558:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1761233737.649558:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233737/real 1761233737] req@00000000fd4aa3a2 x1846780895168256/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233762 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233737.649560:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233737.649564:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6298 00000100:00080000:2.0:1761233737.649566:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233737.649569:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233737.649572:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233737.649574:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895168256/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233762 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233737.649580:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895168576/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233737.649583:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233737.649586:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761233737.649587:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233737.649589:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1761233737.649590:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895168256:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761233737.649591:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233737.649594:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6298 00000100:00080000:2.0:1761233737.649595:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233737.649596:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895168320:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:2.0:1761233737.649598:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233737.649601:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233737.649609:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895168640/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233737.649616:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233737.649624:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233737/real 1761233737] req@000000006c1041ba x1846780895168320/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233762 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233737.649635:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895168320/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233762 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233737.649643:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233737.649645:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233737.649648:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895168320:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233737.649675:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895168384:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233737.649693:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233737.649699:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233737/real 1761233737] req@000000006d8c3eb0 x1846780895168384/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233762 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233737.649706:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895168384/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233762 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233737.649711:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233737.649712:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233737.649714:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895168384:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233737.649717:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895168448:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233737.649726:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233737.649730:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233737/real 1761233737] req@0000000088939481 x1846780895168448/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233762 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233737.649737:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895168448/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233762 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233737.649741:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233737.649742:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233737.649743:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895168448:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233737.649747:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895168512:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233737.649757:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233737.649761:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233737/real 1761233737] req@000000006ff1fdbc x1846780895168512/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233762 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233737.649767:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895168512/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233762 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233737.649772:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233737.649773:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233737.649774:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895168512:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233737.649777:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895168576:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233737.649786:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233737.649789:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233737/real 1761233737] req@00000000135e9a6e x1846780895168576/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233762 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233737.649795:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895168576/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233762 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233737.649798:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233737.649800:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233737.649801:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895168576:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233737.649804:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895168640:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233737.649813:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233737.649816:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233737/real 1761233737] req@00000000fd3935a6 x1846780895168640/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233762 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233737.649823:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895168640/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233762 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233737.649827:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233737.649828:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233737.649829:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895168640:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233738.417437:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233739.441448:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233740.465384:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233741.489388:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233742.513394:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233742.769330:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233742.769342:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233742.769346:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233742.769358:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6303 00000100:00080000:2.0:1761233742.769361:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233742.769368:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233742.769402:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895168704/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233742.769428:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233742.769432:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233742.769434:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233742.769437:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6303 00000100:00080000:2.0:1761233742.769440:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233742.769444:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233742.769454:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233742.769468:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895168768/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233742.769480:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233742.769484:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233742.769487:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233742.769487:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895168704:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761233742.769494:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6303 00000100:00080000:2.0:1761233742.769498:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233742.769502:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233742.769507:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233742.769521:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895168832/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233742.769530:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233742.769533:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233742.769535:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233742.769538:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6303 00000100:00080000:2.0:1761233742.769541:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233742.769551:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233742.769556:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233742.769568:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895168896/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233742.769575:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233742.769577:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233742.769578:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233742.769581:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6303 00000100:00080000:2.0:1761233742.769582:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233742.769585:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233742.769588:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233742.769596:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895168960/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233742.769601:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233742.769603:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233742.769604:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233742.769607:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6303 00000100:00080000:2.0:1761233742.769608:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233742.769613:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233742.769613:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895168768:192.168.201.157@tcp:38:kworker/u8:2.0 00000008:00100000:2.0:1761233742.769619:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233742.769629:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895169024/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233742.769637:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233742.769639:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233742.769641:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233742.769644:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6303 00000100:00080000:2.0:1761233742.769647:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233742.769647:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895168832:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00080000:2.0:1761233742.769654:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233742.769660:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233742.769675:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895169088/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233742.769679:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895168896:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233742.769713:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895168960:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233742.769739:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895169024:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233742.769764:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895169088:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233743.537414:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233744.561388:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233745.585381:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:1.0:1761233745.841515:0:2210:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:1.0:1761233745.841563:0:2210:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1761233745.841590:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233745.841630:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233745.841641:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233745.841649:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233745.841657:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233745.841667:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:1.0:1761233745.841677:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:3.0:1761233745.841694:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233742/real 1761233745] req@000000007091f547 x1846780895169024/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233767 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233745.841727:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895169024/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233767 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233745.841745:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233745.841749:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233745.841754:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895169024:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233745.841762:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233742/real 1761233745] req@0000000064a2cea0 x1846780895169088/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233767 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233745.841781:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895169088/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233767 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233745.841788:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233745.841791:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233745.841794:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895169088:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233745.841800:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233742/real 1761233745] req@0000000033765045 x1846780895168768/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233767 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233745.841813:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895168768/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233767 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233745.841820:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233745.841822:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233745.841825:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895168768:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233745.841831:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233742/real 1761233745] req@000000009d315fa5 x1846780895168832/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233767 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233745.841849:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895168832/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233767 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233745.841856:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233745.841859:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233745.841862:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895168832:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233745.841868:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233742/real 1761233745] req@0000000040d5f361 x1846780895168896/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233767 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233745.841879:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895168896/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233767 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233745.841886:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233745.841888:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233745.841890:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895168896:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233745.841896:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233742/real 1761233745] req@000000007dcf84ae x1846780895168960/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233767 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233745.841904:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895168960/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233767 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233745.841911:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233745.841914:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233745.841918:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895168960:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233745.841923:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233742/real 1761233745] req@000000004f29ca26 x1846780895168704/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233767 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233745.841939:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895168704/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233767 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233745.841944:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233745.841947:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233745.841950:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895168704:192.168.201.157@tcp:250:kworker/u8:2.0 00010000:00010000:3.0:1761233746.609401:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233747.633369:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233747.889339:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233747.889350:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233747.889355:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233747.889366:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6308 00000100:00080000:0.0:1761233747.889370:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233747.889377:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233747.889409:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895169152/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233747.889438:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233747.889442:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233747.889444:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233747.889448:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6308 00000100:00080000:0.0:1761233747.889450:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233747.889453:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233747.889463:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233747.889475:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895169216/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233747.889485:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233747.889489:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1761233747.889489:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895169152:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233747.889493:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233747.889499:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6308 00000100:00080000:0.0:1761233747.889501:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233747.889505:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233747.889510:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233747.889522:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895169280/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233747.889532:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233747.889535:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233747.889536:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233747.889539:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6308 00000100:00080000:0.0:1761233747.889541:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233747.889546:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233747.889549:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233747.889555:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233747.889561:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895169344/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233747.889571:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233747.889575:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233747.889576:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233747.889582:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6308 00000100:00100000:3.0:1761233747.889582:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233747/real 1761233747] req@00000000d67bb589 x1846780895169152/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233772 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233747.889587:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233747.889595:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233747.889601:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233747.889609:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895169152/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233772 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233747.889614:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895169408/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233747.889627:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233747.889630:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233747.889633:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233747.889635:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233747.889638:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233747.889642:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895169152:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233747.889646:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6308 00000100:00080000:0.0:1761233747.889650:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233747.889655:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233747.889659:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233747.889670:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895169472/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233747.889674:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895169216:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:0.0:1761233747.889686:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233747.889691:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233747.889693:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233747.889698:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6308 00000100:00080000:0.0:1761233747.889701:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233747.889705:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233747.889710:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233747.889712:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233747.889725:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895169536/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233747.889728:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233747/real 1761233747] req@000000009d1c7f73 x1846780895169216/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233772 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233747.889757:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895169216/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233772 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233747.889772:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233747.889774:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233747.889779:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895169216:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233747.889787:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895169280:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233747.889809:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233747.889817:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233747/real 1761233747] req@000000004130b6bd x1846780895169280/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233772 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233747.889827:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895169280/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233772 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233747.889834:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233747.889837:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233747.889839:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895169280:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233747.889853:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895169344:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233747.889873:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233747.889883:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233747/real 1761233747] req@00000000ef655201 x1846780895169344/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233772 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233747.889893:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895169344/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233772 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233747.889901:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233747.889903:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233747.889905:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895169344:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233747.889911:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895169408:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233747.889929:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233747.889937:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233747/real 1761233747] req@000000006d61dd06 x1846780895169408/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233772 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233747.889947:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895169408/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233772 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233747.889953:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233747.889955:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233747.889958:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895169408:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233747.889977:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895169472:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233747.889995:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233747.890003:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233747/real 1761233747] req@00000000efa29eb9 x1846780895169472/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233772 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233747.890013:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895169472/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233772 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233747.890020:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233747.890022:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233747.890025:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895169472:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233747.890031:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895169536:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233747.890083:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233747.890094:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233747/real 1761233747] req@000000001f9ec359 x1846780895169536/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233772 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233747.890105:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895169536/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233772 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233747.890113:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233747.890115:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233747.890118:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895169536:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233748.657420:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233749.681353:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233750.705355:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233751.729356:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233752.753400:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233753.009307:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233753.009316:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233753.009319:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233753.009329:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6313 00000100:00080000:0.0:1761233753.009332:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233753.009339:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233753.009369:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895169600/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233753.009391:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233753.009394:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233753.009395:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233753.009397:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6313 00000100:00080000:0.0:1761233753.009399:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233753.009401:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233753.009407:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233753.009414:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895169664/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233753.009420:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233753.009423:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233753.009424:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233753.009428:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6313 00000100:00080000:0.0:1761233753.009429:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233753.009431:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233753.009434:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233753.009439:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895169728/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233753.009445:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233753.009448:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233753.009450:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233753.009450:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895169600:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233753.009456:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6313 00000100:00080000:0.0:1761233753.009459:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233753.009461:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233753.009464:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233753.009471:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895169792/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233753.009476:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233753.009479:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233753.009479:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233753.009482:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6313 00000100:00080000:0.0:1761233753.009484:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233753.009486:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233753.009489:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233753.009495:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895169856/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233753.009499:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233753.009501:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233753.009502:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233753.009504:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6313 00000100:00080000:0.0:1761233753.009505:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233753.009508:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233753.009510:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233753.009516:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895169920/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233753.009526:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1761233753.009529:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233753.009530:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233753.009532:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233753.009537:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6313 00000100:00080000:0.0:1761233753.009538:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233753.009540:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233753.009543:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233753.009551:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895169984/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233753.009557:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233753/real 1761233753] req@000000005593c994 x1846780895169600/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233778 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233753.009575:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895169600/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233778 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233753.009582:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233753.009585:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233753.009587:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895169600:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233753.009593:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895169664:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233753.009612:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233753.009620:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233753/real 1761233753] req@00000000c6e06344 x1846780895169664/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233778 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233753.009637:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895169664/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233778 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233753.009644:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233753.009647:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233753.009650:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895169664:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233753.009676:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895169728:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233753.009699:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233753.009708:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233753/real 1761233753] req@000000004862b808 x1846780895169728/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233778 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233753.009723:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895169728/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233778 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233753.009728:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233753.009730:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233753.009731:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895169728:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233753.009735:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895169792:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233753.009748:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233753.009753:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233753/real 1761233753] req@000000006f2efc8b x1846780895169792/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233778 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233753.009759:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895169792/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233778 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233753.009763:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233753.009765:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233753.009767:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895169792:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233753.009770:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895169856:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233753.009780:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233753.009784:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233753/real 1761233753] req@00000000eca9ddc2 x1846780895169856/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233778 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233753.009790:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895169856/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233778 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233753.009794:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233753.009795:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233753.009797:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895169856:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233753.009800:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895169920:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233753.009809:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233753.009813:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233753/real 1761233753] req@0000000034d49f59 x1846780895169920/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233778 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233753.009819:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895169920/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233778 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233753.009823:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233753.009824:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233753.009825:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895169920:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233753.009828:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895169984:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233753.009839:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233753.009843:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233753/real 1761233753] req@000000005beaae31 x1846780895169984/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233778 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233753.009849:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895169984/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233778 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233753.009853:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233753.009855:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233753.009856:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895169984:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233753.777361:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233754.801407:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233755.825423:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233756.849434:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233757.873436:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233758.129319:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233758.129331:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233758.129335:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233758.129348:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6319 00000100:00080000:0.0:1761233758.129352:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233758.129360:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233758.129396:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000078e7052d x1846780895170048/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233758.129424:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233758.129427:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233758.129429:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233758.129433:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6319 00000100:00080000:0.0:1761233758.129435:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233758.129438:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233758.129451:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233758.129465:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000addc3db8 x1846780895170112/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233758.129468:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895170048:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233758.129482:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233758.129485:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233758.129487:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233758.129492:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6319 00000100:00080000:0.0:1761233758.129494:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233758.129497:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233758.129503:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233758.129516:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895170176/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233758.129525:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233758.129528:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233758.129530:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233758.129533:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6319 00000100:00080000:0.0:1761233758.129535:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1761233758.129539:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233758.129543:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233758.129549:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233758.129562:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895170240/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233758.129567:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233758/real 1761233758] req@0000000078e7052d x1846780895170048/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233783 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233758.129575:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233758.129582:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233758.129584:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233758.129591:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6319 00000100:00080000:0.0:1761233758.129593:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233758.129598:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000078e7052d x1846780895170048/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233783 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233758.129600:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233758.129607:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761233758.129614:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233758.129619:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233758.129622:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895170304/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233758.129626:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895170048:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233758.129638:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233758.129642:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233758.129644:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233758.129648:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6319 00000100:00080000:0.0:1761233758.129650:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233758.129654:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233758.129659:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233758.129659:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895170112:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761233758.129676:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895170368/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233758.129686:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233758.129689:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233758.129692:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1761233758.129693:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233758.129700:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6319 00000100:00080000:0.0:1761233758.129703:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233758.129707:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233758/real 1761233758] req@00000000addc3db8 x1846780895170112/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233783 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233758.129709:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233758.129716:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233758.129723:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000addc3db8 x1846780895170112/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233783 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233758.129730:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1761233758.129731:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895170432/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233758.129736:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233758.129744:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895170112:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233758.129758:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895170176:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233758.129787:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233758.129796:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233758/real 1761233758] req@000000005beaae31 x1846780895170176/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233783 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233758.129807:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895170176/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233783 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233758.129815:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233758.129817:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233758.129820:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895170176:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233758.129825:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895170240:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233758.129842:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233758.129848:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233758/real 1761233758] req@0000000034d49f59 x1846780895170240/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233783 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233758.129858:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895170240/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233783 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233758.129865:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233758.129867:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233758.129870:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895170240:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233758.129875:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895170304:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233758.129921:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233758.129929:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233758/real 1761233758] req@00000000eca9ddc2 x1846780895170304/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233783 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233758.129940:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895170304/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233783 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233758.129948:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233758.129950:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233758.129953:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895170304:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233758.129973:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895170368:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233758.129993:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233758.130001:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233758/real 1761233758] req@000000006f2efc8b x1846780895170368/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233783 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233758.130012:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895170368/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233783 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233758.130020:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233758.130022:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233758.130024:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895170368:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233758.130029:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895170432:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233758.130047:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233758.130052:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233758/real 1761233758] req@000000004862b808 x1846780895170432/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233783 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233758.130062:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895170432/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233783 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233758.130069:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233758.130071:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233758.130074:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895170432:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233758.897389:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233759.921411:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233760.945427:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233761.969401:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233762.993417:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233763.249296:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233763.249305:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233763.249308:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233763.249318:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6324 00000100:00080000:0.0:1761233763.249321:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233763.249326:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233763.249358:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895170496/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233763.249384:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233763.249386:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233763.249387:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233763.249390:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6324 00000100:00080000:0.0:1761233763.249391:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233763.249393:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233763.249400:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233763.249407:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895170560/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233763.249412:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233763.249415:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233763.249416:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233763.249418:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6324 00000100:00080000:0.0:1761233763.249420:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233763.249422:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233763.249427:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233763.249432:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895170624/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233763.249438:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233763.249440:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233763.249442:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233763.249442:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895170496:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233763.249448:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6324 00000100:00080000:0.0:1761233763.249450:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233763.249451:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233763.249454:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233763.249462:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895170688/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233763.249467:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233763.249469:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233763.249470:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233763.249473:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6324 00000100:00080000:0.0:1761233763.249474:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233763.249476:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233763.249479:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233763.249485:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895170752/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233763.249489:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233763.249490:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233763.249492:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233763.249496:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6324 00000100:00080000:0.0:1761233763.249497:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233763.249499:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233763.249503:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233763.249510:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895170816/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233763.249512:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233763.249518:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233763.249520:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233763.249521:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233763.249525:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6324 00000100:00080000:0.0:1761233763.249527:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233763.249531:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233763.249534:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233763.249537:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233763/real 1761233763] req@00000000c6e06344 x1846780895170496/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233788 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233763.249548:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895170880/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233763.249558:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895170496/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233788 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233763.249570:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233763.249574:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233763.249579:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895170496:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233763.249584:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895170560:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233763.249602:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233763.249609:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233763/real 1761233763] req@000000005593c994 x1846780895170560/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233788 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233763.249616:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895170560/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233788 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233763.249621:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233763.249622:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233763.249624:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895170560:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233763.249644:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895170624:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233763.249655:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233763.249659:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233763/real 1761233763] req@000000001f9ec359 x1846780895170624/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233788 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233763.249665:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895170624/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233788 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233763.249670:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233763.249671:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233763.249673:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895170624:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233763.249675:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895170688:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233763.249684:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233763.249688:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233763/real 1761233763] req@00000000efa29eb9 x1846780895170688/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233788 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233763.249693:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895170688/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233788 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233763.249702:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233763.249703:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233763.249705:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895170688:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233763.249708:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895170752:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233763.249718:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233763.249722:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233763/real 1761233763] req@000000006d61dd06 x1846780895170752/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233788 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233763.249728:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895170752/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233788 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233763.249732:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233763.249734:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233763.249735:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895170752:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233763.249738:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895170816:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233763.249747:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233763.249751:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233763/real 1761233763] req@00000000ef655201 x1846780895170816/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233788 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233763.249757:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895170816/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233788 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233763.249761:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233763.249762:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233763.249764:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895170816:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233763.249768:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895170880:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233763.249777:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233763.249781:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233763/real 1761233763] req@000000004130b6bd x1846780895170880/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233788 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233763.249787:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895170880/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233788 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233763.249791:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233763.249792:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233763.249794:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895170880:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233764.017396:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233765.041400:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233766.065456:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233767.089406:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233768.113381:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233768.369313:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233768.369322:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233768.369325:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233768.369336:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6329 00000100:00080000:0.0:1761233768.369339:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233768.369344:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233768.369377:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895170944/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233768.369410:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233768.369414:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233768.369415:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233768.369417:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6329 00000100:00080000:0.0:1761233768.369418:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233768.369421:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233768.369428:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233768.369435:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895171008/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233768.369441:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233768.369443:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233768.369444:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233768.369447:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6329 00000100:00080000:0.0:1761233768.369449:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233768.369451:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233768.369460:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233768.369466:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895170944:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233768.369469:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895171072/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233768.369480:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233768.369481:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233768.369483:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233768.369485:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6329 00000100:00080000:0.0:1761233768.369486:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233768.369499:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233768.369502:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233768.369511:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895171136/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233768.369516:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233768.369521:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233768.369522:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233768.369525:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6329 00000100:00080000:0.0:1761233768.369526:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233768.369529:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233768.369533:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233768.369544:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895171200/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233768.369552:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233768.369557:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1761233768.369557:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233768.369561:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233768.369566:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6329 00000100:00080000:0.0:1761233768.369569:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233768.369573:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233768.369578:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233768.369581:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233768/real 1761233768] req@000000009d1c7f73 x1846780895170944/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233793 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233768.369592:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895171264/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233768.369597:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895170944/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233793 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233768.369618:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233768.369619:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233768.369624:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233768.369625:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1761233768.369628:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895170944:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233768.369629:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233768.369636:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895171008:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:0.0:1761233768.369637:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6329 00000100:00080000:0.0:1761233768.369643:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233768.369648:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233768.369653:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233768.369659:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233768.369667:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233768/real 1761233768] req@00000000d67bb589 x1846780895171008/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233793 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233768.369670:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895171328/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233768.369692:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895171008/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233793 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233768.369698:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233768.369700:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233768.369704:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895171008:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233768.369724:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895171072:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233768.369755:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233768.369760:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233768/real 1761233768] req@00000000a4a2c44a x1846780895171072/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233793 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233768.369767:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895171072/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233793 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233768.369772:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233768.369773:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233768.369775:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895171072:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233768.369778:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895171136:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233768.369788:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233768.369792:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233768/real 1761233768] req@000000004a74da19 x1846780895171136/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233793 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233768.369798:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895171136/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233793 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233768.369803:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233768.369804:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233768.369806:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895171136:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233768.369809:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895171200:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233768.369819:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233768.369822:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233768/real 1761233768] req@000000002d1b8ef7 x1846780895171200/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233793 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233768.369828:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895171200/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233793 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233768.369832:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233768.369834:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233768.369835:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895171200:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233768.369839:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895171264:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233768.369847:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233768.369851:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233768/real 1761233768] req@000000007bd1ec64 x1846780895171264/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233793 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233768.369856:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895171264/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233793 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233768.369861:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233768.369862:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233768.369863:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895171264:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233768.369867:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895171328:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233768.369876:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233768.369880:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233768/real 1761233768] req@000000001db96463 x1846780895171328/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233793 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233768.369886:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895171328/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233793 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233768.369890:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233768.369891:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233768.369893:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895171328:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233769.137384:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233770.161392:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233771.185349:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233772.209437:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233773.233416:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233773.489318:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233773.489330:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233773.489334:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233773.489347:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6334 00000100:00080000:0.0:1761233773.489351:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233773.489358:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233773.489397:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895171392/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233773.489426:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233773.489430:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233773.489431:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233773.489435:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6334 00000100:00080000:0.0:1761233773.489437:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233773.489441:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233773.489450:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233773.489459:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895171456/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233773.489466:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233773.489468:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233773.489471:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233773.489474:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6334 00000100:00080000:0.0:1761233773.489475:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233773.489478:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233773.489480:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233773.489488:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895171520/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233773.489494:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895171392:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233773.489496:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233773.489502:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233773.489504:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233773.489507:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6334 00000100:00080000:0.0:1761233773.489509:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233773.489511:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233773.489513:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233773.489521:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895171584/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233773.489526:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233773.489528:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233773.489529:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233773.489532:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6334 00000100:00080000:0.0:1761233773.489534:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233773.489536:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233773.489539:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233773.489545:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895171648/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233773.489550:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233773.489552:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233773.489554:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233773.489556:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6334 00000100:00080000:0.0:1761233773.489558:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233773.489560:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233773.489562:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233773.489570:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895171712/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233773.489572:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233773.489578:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233773.489580:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233773.489582:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233773.489584:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6334 00000100:00080000:0.0:1761233773.489586:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233773.489588:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233773.489591:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233773.489598:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895171776/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233773.489602:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233773/real 1761233773] req@00000000f3ed2588 x1846780895171392/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233798 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233773.489622:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f3ed2588 x1846780895171392/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233798 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233773.489632:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233773.489635:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233773.489639:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895171392:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233773.489646:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895171456:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233773.489685:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233773.489694:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233773/real 1761233773] req@000000005d83493f x1846780895171456/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233798 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233773.489704:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005d83493f x1846780895171456/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233798 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233773.489711:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233773.489713:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233773.489717:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895171456:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233773.489741:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895171520:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233773.489758:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233773.489765:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233773/real 1761233773] req@000000001db96463 x1846780895171520/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233798 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233773.489774:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895171520/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233798 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233773.489780:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233773.489782:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233773.489785:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895171520:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233773.489790:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895171584:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233773.489804:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233773.489811:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233773/real 1761233773] req@000000007bd1ec64 x1846780895171584/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233798 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233773.489821:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895171584/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233798 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233773.489828:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233773.489830:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233773.489833:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895171584:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233773.489838:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895171648:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233773.489854:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233773.489861:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233773/real 1761233773] req@000000002d1b8ef7 x1846780895171648/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233798 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233773.489871:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895171648/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233798 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233773.489877:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233773.489880:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233773.489882:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895171648:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233773.489887:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895171712:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233773.489903:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233773.489910:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233773/real 1761233773] req@000000004a74da19 x1846780895171712/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233798 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233773.489919:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895171712/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233798 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233773.489925:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233773.489927:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233773.489930:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895171712:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233773.489935:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895171776:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233773.489950:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233773.489957:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233773/real 1761233773] req@00000000a4a2c44a x1846780895171776/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233798 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233773.489966:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895171776/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233798 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233773.489972:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233773.489974:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233773.489976:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895171776:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233774.257395:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233775.281399:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233776.305441:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233777.329340:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233778.353398:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233778.609348:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233778.609366:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233778.609372:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233778.609386:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6339 00000100:00080000:1.0:1761233778.609392:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233778.609401:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233778.609447:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ba3340f x1846780895171840/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233778.609489:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233778.609493:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233778.609495:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233778.609499:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6339 00000100:00080000:1.0:1761233778.609502:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233778.609506:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233778.609517:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233778.609530:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c652d0ad x1846780895171904/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233778.609538:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233778.609542:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233778.609544:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233778.609548:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6339 00000100:00080000:1.0:1761233778.609551:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233778.609555:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233778.609559:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233778.609566:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895171840:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233778.609604:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000568f2a96 x1846780895171968/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233778.609614:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233778.609618:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233778.609620:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233778.609623:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6339 00000100:00080000:1.0:1761233778.609626:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233778.609630:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233778.609635:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233778.609645:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000053e52331 x1846780895172032/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233778.609653:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233778.609657:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233778.609659:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233778.609663:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6339 00000100:00080000:1.0:1761233778.609665:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233778.609668:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233778.609672:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233778.609677:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233778.609685:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090ac513e x1846780895172096/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233778.609694:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233778.609698:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233778.609700:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233778.609703:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6339 00000100:00080000:1.0:1761233778.609706:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233778.609710:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233778/real 1761233778] req@000000007ba3340f x1846780895171840/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233803 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233778.609712:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233778.609720:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233778.609732:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000c736bb2 x1846780895172160/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233778.609740:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007ba3340f x1846780895171840/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233803 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233778.609743:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233778.609751:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233778.609754:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233778.609760:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6339 00000100:00080000:3.0:1761233778.609761:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233778.609763:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233778.609767:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:1.0:1761233778.609769:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233778.609774:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233778.609776:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895171840:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233778.609787:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a95774fb x1846780895172224/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233778.609790:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895171904:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233778.609830:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233778.609839:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233778/real 1761233778] req@00000000c652d0ad x1846780895171904/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233803 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233778.609851:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c652d0ad x1846780895171904/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233803 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233778.609860:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233778.609862:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233778.609866:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895171904:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233778.609895:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895171968:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233778.609913:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233778.609920:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233778/real 1761233778] req@00000000568f2a96 x1846780895171968/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233803 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233778.609931:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000568f2a96 x1846780895171968/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233803 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233778.609939:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233778.609941:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233778.609944:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895171968:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233778.609949:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895172032:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233778.609964:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233778.609969:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233778/real 1761233778] req@0000000053e52331 x1846780895172032/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233803 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233778.609979:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000053e52331 x1846780895172032/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233803 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233778.609987:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233778.609990:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233778.609993:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895172032:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233778.609998:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895172096:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233778.610012:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233778.610018:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233778/real 1761233778] req@0000000090ac513e x1846780895172096/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233803 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233778.610028:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090ac513e x1846780895172096/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233803 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233778.610036:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233778.610038:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233778.610041:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895172096:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233778.610046:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895172160:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233778.610061:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233778.610067:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233778/real 1761233778] req@000000000c736bb2 x1846780895172160/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233803 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233778.610077:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000000c736bb2 x1846780895172160/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233803 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233778.610084:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233778.610086:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233778.610089:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895172160:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233778.610095:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895172224:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233778.610109:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233778.610115:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233778/real 1761233778] req@00000000a95774fb x1846780895172224/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233803 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233778.610125:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a95774fb x1846780895172224/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233803 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233778.610133:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233778.610135:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233778.610137:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895172224:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233779.377445:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233780.401366:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233781.425416:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233782.449311:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233783.473369:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233783.729326:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233783.729339:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233783.729342:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233783.729354:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6344 00000100:00080000:0.0:1761233783.729358:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233783.729366:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233783.729401:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895172288/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233783.729429:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233783.729433:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233783.729434:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233783.729437:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6344 00000100:00080000:0.0:1761233783.729439:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233783.729443:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233783.729453:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233783.729464:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895172352/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233783.729473:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233783.729477:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233783.729479:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233783.729484:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6344 00000100:00080000:0.0:1761233783.729498:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233783.729502:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233783.729508:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233783.729512:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895172288:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761233783.729521:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895172416/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233783.729540:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233783.729544:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233783.729545:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233783.729549:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6344 00000100:00080000:0.0:1761233783.729552:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233783.729555:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233783.729559:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233783.729570:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895172480/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233783.729577:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233783.729580:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233783.729582:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233783.729585:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6344 00000100:00080000:0.0:1761233783.729587:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233783.729593:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:3.0:1761233783.729598:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:0.0:1761233783.729600:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233783.729616:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895172544/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233783.729625:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233783.729628:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233783/real 1761233783] req@00000000d67bb589 x1846780895172288/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233808 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233783.729630:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233783.729632:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233783.729640:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6344 00000100:00080000:0.0:1761233783.729645:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233783.729650:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233783.729655:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233783.729656:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895172288/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233808 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233783.729666:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1761233783.729668:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895172608/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233783.729670:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233783.729676:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895172288:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233783.729685:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895172352:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:0.0:1761233783.729698:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233783.729702:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233783.729705:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233783.729709:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6344 00000100:00080000:0.0:1761233783.729713:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1761233783.729714:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233783.729722:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233783.729725:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233783/real 1761233783] req@000000009d1c7f73 x1846780895172352/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233808 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:0.0:1761233783.729729:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233783.729736:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895172352/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233808 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233783.729743:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1761233783.729745:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895172672/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233783.729746:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233783.729752:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895172352:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233783.729782:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895172416:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233783.729797:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233783.729801:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233783/real 1761233783] req@000000004130b6bd x1846780895172416/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233808 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233783.729808:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895172416/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233808 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233783.729812:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233783.729814:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233783.729816:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895172416:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233783.729819:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895172480:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233783.729828:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233783.729831:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233783/real 1761233783] req@00000000ef655201 x1846780895172480/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233808 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233783.729837:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895172480/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233808 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233783.729842:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233783.729843:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233783.729845:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895172480:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233783.729848:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895172544:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233783.729858:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233783.729861:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233783/real 1761233783] req@000000006d61dd06 x1846780895172544/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233808 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233783.729866:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895172544/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233808 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233783.729871:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233783.729872:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233783.729874:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895172544:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233783.729877:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895172608:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233783.729886:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233783.729890:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233783/real 1761233783] req@00000000efa29eb9 x1846780895172608/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233808 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233783.729895:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895172608/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233808 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233783.729905:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233783.729906:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233783.729908:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895172608:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233783.729912:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895172672:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233783.729923:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233783.729927:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233783/real 1761233783] req@000000001f9ec359 x1846780895172672/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233808 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233783.729933:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895172672/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233808 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233783.729937:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233783.729939:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233783.729940:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895172672:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233784.497362:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233785.521394:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233786.545383:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233787.569348:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233788.593326:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233788.849289:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233788.849298:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233788.849302:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233788.849310:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6349 00000100:00080000:0.0:1761233788.849314:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233788.849319:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233788.849348:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895172736/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233788.849370:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233788.849374:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233788.849376:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233788.849380:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6349 00000100:00080000:0.0:1761233788.849382:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233788.849385:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233788.849397:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233788.849428:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895172736:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761233788.849442:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895172800/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233788.849454:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233788.849465:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233788.849467:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233788.849471:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6349 00000100:00080000:0.0:1761233788.849477:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233788.849481:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233788.849487:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233788.849491:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233788.849503:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895172864/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233788.849512:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233788.849513:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233788/real 1761233788] req@000000005593c994 x1846780895172736/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233813 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233788.849520:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233788.849522:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233788.849528:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6349 00000100:00080000:0.0:1761233788.849531:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233788.849533:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895172736/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233813 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233788.849537:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233788.849545:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761233788.849547:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233788.849550:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233788.849555:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895172736:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761233788.849559:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895172928/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233788.849567:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233788.849570:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233788.849572:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233788.849576:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6349 00000100:00080000:0.0:1761233788.849579:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233788.849580:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895172800:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:0.0:1761233788.849586:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233788.849592:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233788.849599:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233788.849608:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895172992/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233788.849609:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233788/real 1761233788] req@00000000c6e06344 x1846780895172800/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233813 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233788.849642:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895172800/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233813 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233788.849645:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233788.849652:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233788.849653:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233788.849655:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233788.849658:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233788.849659:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895172800:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:0.0:1761233788.849669:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6349 00000100:00100000:3.0:1761233788.849669:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895172864:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1761233788.849674:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233788.849682:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:3.0:1761233788.849686:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:0.0:1761233788.849688:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233788.849692:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233788/real 1761233788] req@000000004862b808 x1846780895172864/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233813 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233788.849705:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895172864/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233813 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233788.849706:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895173056/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233788.849722:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233788.849725:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233788.849727:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233788.849729:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895172864:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1761233788.849734:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233788.849737:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233788.849737:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895172928:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:0.0:1761233788.849746:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6349 00000100:00080000:0.0:1761233788.849749:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233788.849753:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:3.0:1761233788.849756:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:0.0:1761233788.849759:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233788.849763:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233788/real 1761233788] req@000000006f2efc8b x1846780895172928/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233813 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233788.849782:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895173120/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233788.849782:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895172928/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233813 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233788.849810:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233788.849814:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233788.849818:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895172928:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233788.849838:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895172992:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233788.849851:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233788.849856:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233788/real 1761233788] req@00000000eca9ddc2 x1846780895172992/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233813 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233788.849861:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895172992/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233813 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233788.849865:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233788.849866:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233788.849868:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895172992:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233788.849871:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895173056:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233788.849880:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233788.849884:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233788/real 1761233788] req@0000000034d49f59 x1846780895173056/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233813 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233788.849890:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895173056/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233813 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233788.849894:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233788.849895:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233788.849897:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895173056:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233788.849900:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895173120:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233788.849909:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233788.849912:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233788/real 1761233788] req@000000005beaae31 x1846780895173120/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233813 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233788.849918:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895173120/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233813 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233788.849922:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233788.849924:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233788.849927:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895173120:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233789.617458:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233790.641364:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233791.665484:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233792.689526:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233793.713523:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233793.969392:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233793.969406:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233793.969411:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233793.969423:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6354 00000100:00080000:1.0:1761233793.969427:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233793.969435:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233793.969474:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cb5be845 x1846780895173184/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233793.969505:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233793.969509:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233793.969511:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233793.969514:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6354 00000100:00080000:1.0:1761233793.969517:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233793.969521:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233793.969531:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233793.969542:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000880f16b4 x1846780895173248/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233793.969552:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233793.969555:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233793.969558:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233793.969561:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6354 00000100:00080000:1.0:1761233793.969564:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233793.969568:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233793.969573:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233793.969576:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895173184:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233793.969584:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000038209f5 x1846780895173312/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233793.969595:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233793.969598:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233793.969600:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233793.969605:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6354 00000100:00080000:1.0:1761233793.969607:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233793.969611:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233793.969615:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233793.969626:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d66930c2 x1846780895173376/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233793.969634:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233793.969638:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233793.969640:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233793.969643:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6354 00000100:00080000:1.0:1761233793.969645:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233793.969649:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233793.969654:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233793.969655:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233793.969666:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ffe86d04 x1846780895173440/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233793.969675:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233793.969678:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233793.969680:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233793.969684:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233793/real 1761233793] req@00000000cb5be845 x1846780895173184/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233818 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233793.969688:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6354 00000100:00080000:1.0:1761233793.969693:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233793.969698:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233793.969704:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233793.969709:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cb5be845 x1846780895173184/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233818 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233793.969718:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df5b2270 x1846780895173504/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233793.969723:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233793.969728:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233793.969735:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895173184:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233793.969748:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233793.969748:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895173248:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:1.0:1761233793.969754:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233793.969757:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233793.969762:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6354 00000100:00080000:1.0:1761233793.969764:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233793.969768:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233793.969773:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233793.969780:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233793.969787:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df0b94a8 x1846780895173568/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233793.969792:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233793/real 1761233793] req@00000000880f16b4 x1846780895173248/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233818 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233793.969811:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000880f16b4 x1846780895173248/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233818 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233793.969819:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233793.969822:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233793.969826:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895173248:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233793.969854:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895173312:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233793.969872:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233793.969881:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233793/real 1761233793] req@00000000038209f5 x1846780895173312/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233818 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233793.969890:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000038209f5 x1846780895173312/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233818 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233793.969897:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233793.969899:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233793.969902:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895173312:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233793.969908:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895173376:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233793.969924:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233793.969931:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233793/real 1761233793] req@00000000d66930c2 x1846780895173376/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233818 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233793.969940:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d66930c2 x1846780895173376/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233818 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233793.969947:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233793.969949:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233793.969952:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895173376:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233793.969957:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895173440:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233793.969973:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233793.969980:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233793/real 1761233793] req@00000000ffe86d04 x1846780895173440/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233818 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233793.969989:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ffe86d04 x1846780895173440/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233818 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233793.969995:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233793.969997:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233793.970000:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895173440:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233793.970005:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895173504:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233793.970021:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233793.970027:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233793/real 1761233793] req@00000000df5b2270 x1846780895173504/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233818 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233793.970036:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df5b2270 x1846780895173504/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233818 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233793.970042:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233793.970044:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233793.970047:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895173504:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233793.970052:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895173568:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233793.970067:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233793.970073:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233793/real 1761233793] req@00000000df0b94a8 x1846780895173568/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233818 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233793.970083:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df0b94a8 x1846780895173568/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233818 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233793.970089:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233793.970091:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233793.970094:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895173568:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233794.737437:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233795.761360:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233796.785458:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233797.809380:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233798.833426:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233799.089351:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233799.089364:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233799.089368:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233799.089380:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6360 00000100:00080000:1.0:1761233799.089385:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233799.089392:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233799.089428:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac65ecb7 x1846780895173632/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233799.089454:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233799.089459:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233799.089461:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233799.089466:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6360 00000100:00080000:1.0:1761233799.089469:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233799.089474:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233799.089484:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233799.089498:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006a5e9865 x1846780895173696/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233799.089506:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac65ecb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895173632:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233799.089511:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233799.089516:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233799.089517:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233799.089522:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6360 00000100:00080000:1.0:1761233799.089524:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233799.089528:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233799.089533:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233799.089545:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004361f727 x1846780895173760/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233799.089554:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233799.089557:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233799.089559:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233799.089562:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6360 00000100:00080000:1.0:1761233799.089565:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233799.089568:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233799.089572:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233799.089577:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233799.089585:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df0b94a8 x1846780895173824/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233799.089595:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233799.089598:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233799.089600:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233799.089604:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233799/real 1761233799] req@00000000ac65ecb7 x1846780895173632/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233824 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233799.089606:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6360 00000100:00080000:1.0:1761233799.089610:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233799.089617:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233799.089624:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233799.089631:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac65ecb7 x1846780895173632/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233824 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233799.089637:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df5b2270 x1846780895173888/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233799.089650:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233799.089653:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233799.089655:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:1.0:1761233799.089658:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233799.089662:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233799.089663:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac65ecb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895173632:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233799.089670:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6360 00000100:00080000:1.0:1761233799.089673:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233799.089677:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233799.089681:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233799.089692:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ffe86d04 x1846780895173952/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233799.089695:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a5e9865 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895173696:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:1.0:1761233799.089714:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233799.089719:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233799.089721:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233799.089726:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6360 00000100:00080000:1.0:1761233799.089728:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233799.089734:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:3.0:1761233799.089735:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:1.0:1761233799.089742:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233799.089749:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233799/real 1761233799] req@000000006a5e9865 x1846780895173696/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233824 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233799.089756:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d66930c2 x1846780895174016/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233799.089771:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006a5e9865 x1846780895173696/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233824 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233799.089792:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233799.089795:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233799.089798:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a5e9865 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895173696:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233799.089805:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004361f727 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895173760:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233799.089824:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233799.089831:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233799/real 1761233799] req@000000004361f727 x1846780895173760/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233824 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233799.089842:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004361f727 x1846780895173760/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233824 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233799.089848:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233799.089851:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233799.089853:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004361f727 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895173760:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233799.089859:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895173824:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233799.089874:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233799.089880:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233799/real 1761233799] req@00000000df0b94a8 x1846780895173824/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233824 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233799.089890:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df0b94a8 x1846780895173824/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233824 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233799.089898:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233799.089900:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233799.089902:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895173824:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233799.089908:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895173888:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233799.089924:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233799.089930:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233799/real 1761233799] req@00000000df5b2270 x1846780895173888/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233824 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233799.089940:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df5b2270 x1846780895173888/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233824 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233799.089947:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233799.089950:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233799.089952:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895173888:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233799.089970:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895173952:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233799.089986:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233799.089994:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233799/real 1761233799] req@00000000ffe86d04 x1846780895173952/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233824 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233799.090003:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ffe86d04 x1846780895173952/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233824 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233799.090010:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233799.090012:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233799.090015:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895173952:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233799.090020:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895174016:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233799.090036:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233799.090043:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233799/real 1761233799] req@00000000d66930c2 x1846780895174016/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233824 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233799.090052:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d66930c2 x1846780895174016/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233824 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233799.090059:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233799.090061:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233799.090063:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895174016:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233799.857405:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233800.881427:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233801.905360:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233802.929412:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233803.953344:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233804.209341:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233804.209350:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233804.209353:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233804.209362:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6365 00000100:00080000:0.0:1761233804.209364:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233804.209370:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233804.209405:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000addc3db8 x1846780895174080/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233804.209424:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233804.209427:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233804.209428:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233804.209430:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6365 00000100:00080000:0.0:1761233804.209432:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233804.209434:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233804.209441:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233804.209449:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000078e7052d x1846780895174144/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233804.209455:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233804.209459:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895174080:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233804.209464:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233804.209467:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233804.209474:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6365 00000100:00080000:0.0:1761233804.209476:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233804.209478:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233804.209484:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233804.209492:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895174208/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233804.209498:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233804.209500:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233804.209501:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233804.209504:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6365 00000100:00080000:0.0:1761233804.209505:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233804.209507:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233804.209509:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233804.209515:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895174272/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233804.209520:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233804.209523:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233804.209524:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1761233804.209527:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233804.209531:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6365 00000100:00080000:0.0:1761233804.209535:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233804.209539:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233804.209545:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233804.209550:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233804/real 1761233804] req@00000000addc3db8 x1846780895174080/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233829 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233804.209558:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895174336/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233804.209566:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000addc3db8 x1846780895174080/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233829 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233804.209574:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233804.209578:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233804.209579:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233804.209582:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233804.209583:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233804.209587:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895174080:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233804.209592:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6365 00000100:00080000:0.0:1761233804.209596:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233804.209600:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233804.209605:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233804.209609:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895174144:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761233804.209618:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895174400/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233804.209628:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1761233804.209631:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233804.209633:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233804.209635:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233804.209641:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6365 00000100:00100000:3.0:1761233804.209642:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233804/real 1761233804] req@0000000078e7052d x1846780895174144/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233829 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233804.209643:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233804.209650:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233804.209656:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233804.209659:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000078e7052d x1846780895174144/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233829 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233804.209665:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1761233804.209666:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895174464/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233804.209670:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233804.209676:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895174144:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233804.209683:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895174208:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233804.209698:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233804.209702:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233804/real 1761233804] req@000000005beaae31 x1846780895174208/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233829 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233804.209709:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895174208/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233829 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233804.209713:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233804.209714:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233804.209716:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895174208:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233804.209719:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895174272:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233804.209728:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233804.209732:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233804/real 1761233804] req@0000000034d49f59 x1846780895174272/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233829 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233804.209738:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895174272/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233829 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233804.209742:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233804.209744:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233804.209746:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895174272:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233804.209749:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895174336:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233804.209758:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233804.209762:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233804/real 1761233804] req@00000000eca9ddc2 x1846780895174336/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233829 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233804.209767:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895174336/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233829 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233804.209771:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233804.209773:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233804.209775:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895174336:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233804.209786:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895174400:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233804.209795:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233804.209798:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233804/real 1761233804] req@000000006f2efc8b x1846780895174400/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233829 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233804.209805:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895174400/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233829 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233804.209809:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233804.209810:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233804.209811:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895174400:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233804.209815:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895174464:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233804.209824:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233804.209828:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233804/real 1761233804] req@000000004862b808 x1846780895174464/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233829 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233804.209834:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895174464/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233829 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233804.209838:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233804.209839:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233804.209841:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895174464:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233804.977404:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233806.001403:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233807.025397:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233808.049379:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233809.073446:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233809.329372:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233809.329384:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233809.329389:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233809.329401:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6370 00000100:00080000:1.0:1761233809.329405:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233809.329412:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233809.329456:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000038209f5 x1846780895174528/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233809.329481:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233809.329484:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233809.329486:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233809.329488:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6370 00000100:00080000:1.0:1761233809.329490:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233809.329492:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233809.329501:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233809.329510:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000880f16b4 x1846780895174592/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233809.329516:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233809.329519:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233809.329521:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233809.329529:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895174528:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233809.329532:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6370 00000100:00080000:1.0:1761233809.329535:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233809.329542:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233809.329547:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233809.329559:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cb5be845 x1846780895174656/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233809.329564:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233809.329567:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233809.329569:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233809.329572:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6370 00000100:00080000:1.0:1761233809.329574:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233809.329578:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233809.329583:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233809.329589:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a95774fb x1846780895174720/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233809.329595:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233809.329597:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233809.329598:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233809.329600:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6370 00000100:00080000:1.0:1761233809.329603:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233809.329606:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233809.329609:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233809.329616:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000c736bb2 x1846780895174784/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233809.329620:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233809.329622:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233809.329623:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233809.329629:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6370 00000100:00080000:1.0:1761233809.329631:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233809.329633:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895174592:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:1.0:1761233809.329639:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233809.329646:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233809.329661:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090ac513e x1846780895174848/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233809.329668:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233809.329670:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895174656:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:1.0:1761233809.329673:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233809.329675:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233809.329684:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6370 00000100:00080000:1.0:1761233809.329687:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233809.329692:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233809.329698:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233809.329701:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895174720:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233809.329716:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000053e52331 x1846780895174912/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233809.329734:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895174784:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233809.329759:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895174848:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233809.329776:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895174912:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233810.097343:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233811.121377:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233812.145410:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1761233812.401482:0:2207:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:0.0:1761233812.401529:0:2207:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1761233812.401542:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233812.401573:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233812.401580:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233812.401588:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233812.401598:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233812.401611:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233812.401621:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:3.0:1761233812.401637:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233809/real 1761233812] req@0000000090ac513e x1846780895174848/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233834 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233812.401675:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090ac513e x1846780895174848/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233834 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233812.401685:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233812.401688:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233812.401692:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895174848:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233812.401697:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233809/real 1761233812] req@0000000053e52331 x1846780895174912/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233834 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233812.401705:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000053e52331 x1846780895174912/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233834 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233812.401711:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233812.401713:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233812.401714:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895174912:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233812.401718:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233809/real 1761233812] req@00000000880f16b4 x1846780895174592/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233834 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233812.401726:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000880f16b4 x1846780895174592/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233834 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233812.401730:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233812.401731:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233812.401740:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895174592:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233812.401745:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233809/real 1761233812] req@00000000cb5be845 x1846780895174656/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233834 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233812.401753:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cb5be845 x1846780895174656/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233834 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233812.401757:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233812.401759:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233812.401761:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895174656:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233812.401765:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233809/real 1761233812] req@00000000a95774fb x1846780895174720/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233834 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233812.401771:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a95774fb x1846780895174720/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233834 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233812.401775:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233812.401776:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233812.401778:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895174720:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233812.401782:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233809/real 1761233812] req@000000000c736bb2 x1846780895174784/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233834 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233812.401788:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000000c736bb2 x1846780895174784/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233834 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233812.401792:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233812.401794:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233812.401795:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895174784:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233812.401801:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233809/real 1761233812] req@00000000038209f5 x1846780895174528/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233834 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233812.401812:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000038209f5 x1846780895174528/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233834 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233812.401819:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233812.401823:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233812.401826:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895174528:192.168.201.157@tcp:250:kworker/u8:3.0 00010000:00010000:3.0:1761233813.169394:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233814.193444:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233814.449335:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233814.449347:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233814.449352:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233814.449364:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6375 00000100:00080000:2.0:1761233814.449367:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233814.449374:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233814.449416:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895174976/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233814.449441:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233814.449445:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233814.449446:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233814.449449:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6375 00000100:00080000:2.0:1761233814.449450:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233814.449453:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233814.449463:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233814.449472:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895175040/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233814.449478:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233814.449480:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233814.449483:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233814.449485:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6375 00000100:00080000:2.0:1761233814.449487:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233814.449490:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233814.449493:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233814.449499:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895174976:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761233814.449504:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895175104/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233814.449523:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233814.449526:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233814.449526:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233814.449529:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6375 00000100:00080000:2.0:1761233814.449530:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233814.449533:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233814.449535:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233814.449552:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895175168/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233814.449557:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233814.449559:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233814.449562:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233814.449564:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6375 00000100:00080000:2.0:1761233814.449566:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233814.449569:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233814.449571:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233814.449579:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895175232/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233814.449586:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233814.449589:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233814.449590:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233814.449592:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6375 00000100:00080000:2.0:1761233814.449593:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233814.449595:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233814.449598:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233814.449604:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233814.449608:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895175296/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233814.449614:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233814.449616:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233814.449619:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233814.449623:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6375 00000100:00080000:2.0:1761233814.449625:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233814.449629:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233814.449634:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233814.449638:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233814/real 1761233814] req@000000004f29ca26 x1846780895174976/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233839 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233814.449643:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895175360/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233814.449702:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895174976/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233839 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233814.449717:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233814.449721:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233814.449726:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895174976:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233814.449735:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895175040:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233814.449763:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233814.449772:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233814/real 1761233814] req@000000007dcf84ae x1846780895175040/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233839 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233814.449788:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895175040/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233839 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233814.449809:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233814.449812:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233814.449815:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895175040:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233814.449848:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895175104:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233814.449872:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233814.449880:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233814/real 1761233814] req@0000000040d5f361 x1846780895175104/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233839 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233814.449891:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895175104/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233839 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233814.449898:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233814.449901:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233814.449903:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895175104:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233814.449910:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895175168:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233814.449929:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233814.449936:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233814/real 1761233814] req@000000009d315fa5 x1846780895175168/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233839 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233814.449946:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895175168/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233839 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233814.449953:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233814.449956:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233814.449959:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895175168:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233814.449965:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895175232:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233814.449986:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233814.450001:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233814/real 1761233814] req@0000000033765045 x1846780895175232/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233839 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233814.450047:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895175232/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233839 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233814.450054:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233814.450056:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233814.450058:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895175232:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233814.450064:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895175296:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233814.450085:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233814.450092:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233814/real 1761233814] req@0000000064a2cea0 x1846780895175296/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233839 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233814.450101:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895175296/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233839 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233814.450107:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233814.450110:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233814.450112:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895175296:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233814.450118:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895175360:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233814.450140:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233814.450148:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233814/real 1761233814] req@000000007091f547 x1846780895175360/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233839 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233814.450159:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895175360/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233839 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233814.450165:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233814.450168:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233814.450170:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895175360:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233815.217413:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233816.241422:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233817.265413:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233818.289403:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233819.313380:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233819.569357:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233819.569367:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233819.569370:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233819.569380:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6380 00000100:00080000:1.0:1761233819.569383:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233819.569388:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233819.569423:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000568f2a96 x1846780895175424/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233819.569444:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233819.569446:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233819.569447:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233819.569450:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6380 00000100:00080000:1.0:1761233819.569452:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233819.569454:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233819.569461:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233819.569469:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c652d0ad x1846780895175488/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233819.569476:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233819.569478:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233819.569480:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233819.569480:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895175424:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233819.569484:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6380 00000100:00080000:1.0:1761233819.569487:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233819.569490:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233819.569494:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233819.569501:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ba3340f x1846780895175552/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233819.569506:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233819.569508:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233819.569509:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233819.569511:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6380 00000100:00080000:1.0:1761233819.569513:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233819.569515:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233819.569518:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233819.569525:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d74b2e3c x1846780895175616/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233819.569529:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233819.569532:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233819.569533:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233819.569535:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6380 00000100:00080000:1.0:1761233819.569536:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233819.569539:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233819.569541:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233819.569546:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001aedf9b1 x1846780895175680/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233819.569551:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233819.569553:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233819.569554:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1761233819.569559:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:1.0:1761233819.569560:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6380 00000100:00080000:1.0:1761233819.569563:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233819.569566:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233819.569568:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233819.569574:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a611ee34 x1846780895175744/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233819.569580:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233819.569582:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233819.569583:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233819.569586:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6380 00000100:00080000:1.0:1761233819.569587:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233819.569591:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233819.569591:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233819/real 1761233819] req@00000000568f2a96 x1846780895175424/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233844 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:1.0:1761233819.569595:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233819.569603:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000328368c8 x1846780895175808/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233819.569613:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000568f2a96 x1846780895175424/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233844 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233819.569622:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233819.569624:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233819.569627:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895175424:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233819.569647:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895175488:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233819.569665:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233819.569671:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233819/real 1761233819] req@00000000c652d0ad x1846780895175488/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233844 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233819.569678:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c652d0ad x1846780895175488/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233844 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233819.569683:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233819.569684:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233819.569686:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895175488:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233819.569690:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895175552:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233819.569700:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233819.569703:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233819/real 1761233819] req@000000007ba3340f x1846780895175552/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233844 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233819.569709:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007ba3340f x1846780895175552/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233844 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233819.569713:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233819.569715:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233819.569716:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895175552:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233819.569719:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895175616:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233819.569728:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233819.569731:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233819/real 1761233819] req@00000000d74b2e3c x1846780895175616/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233844 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233819.569737:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d74b2e3c x1846780895175616/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233844 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233819.569742:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233819.569743:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233819.569745:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895175616:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233819.569748:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895175680:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233819.569764:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233819.569768:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233819/real 1761233819] req@000000001aedf9b1 x1846780895175680/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233844 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233819.569774:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001aedf9b1 x1846780895175680/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233844 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233819.569778:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233819.569779:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233819.569781:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895175680:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233819.569784:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895175744:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233819.569793:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233819.569797:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233819/real 1761233819] req@00000000a611ee34 x1846780895175744/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233844 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233819.569803:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a611ee34 x1846780895175744/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233844 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233819.569807:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233819.569809:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233819.569810:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895175744:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233819.569813:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895175808:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233819.569824:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233819.569828:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233819/real 1761233819] req@00000000328368c8 x1846780895175808/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233844 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233819.569834:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000328368c8 x1846780895175808/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233844 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233819.569838:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233819.569839:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233819.569841:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895175808:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233820.337444:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233821.361420:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233822.385432:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233823.409393:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233824.433409:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233824.689358:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233824.689372:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233824.689377:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233824.689389:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6385 00000100:00080000:0.0:1761233824.689393:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233824.689401:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233824.689438:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895175872/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233824.689463:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233824.689467:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233824.689469:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233824.689472:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6385 00000100:00080000:0.0:1761233824.689474:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233824.689479:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233824.689488:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233824.689499:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895175936/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233824.689506:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895175872:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233824.689511:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233824.689515:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233824.689516:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233824.689520:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6385 00000100:00080000:0.0:1761233824.689522:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233824.689525:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233824.689530:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233824.689539:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895176000/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233824.689548:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233824.689552:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233824.689554:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233824.689557:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6385 00000100:00080000:0.0:1761233824.689559:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233824.689562:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233824.689566:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233824.689576:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233824.689588:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895176064/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233824.689598:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233824/real 1761233824] req@00000000c6e06344 x1846780895175872/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233849 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233824.689606:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233824.689611:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233824.689613:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233824.689618:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6385 00000100:00100000:3.0:1761233824.689620:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895175872/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233849 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233824.689622:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233824.689630:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:3.0:1761233824.689631:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233824.689636:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000008:00100000:0.0:1761233824.689637:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233824.689640:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895175872:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233824.689649:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895176128/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233824.689656:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233824.689658:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233824.689659:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233824.689663:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6385 00000100:00100000:3.0:1761233824.689666:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895175936:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:0.0:1761233824.689667:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233824.689673:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233824.689677:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233824.689687:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895176192/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233824.689693:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233824.689699:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233824.689702:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233824.689703:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233824.689705:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233824/real 1761233824] req@000000005593c994 x1846780895175936/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233849 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233824.689708:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6385 00000100:00080000:0.0:1761233824.689712:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233824.689717:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233824.689720:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233824.689722:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895175936/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233849 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233824.689729:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1761233824.689732:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895176256/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233824.689732:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233824.689736:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895175936:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233824.689746:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895176000:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233824.689764:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233824.689769:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233824/real 1761233824] req@000000001f9ec359 x1846780895176000/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233849 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233824.689778:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895176000/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233849 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233824.689784:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233824.689785:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233824.689787:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895176000:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233824.689790:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895176064:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233824.689799:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233824.689803:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233824/real 1761233824] req@00000000efa29eb9 x1846780895176064/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233849 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233824.689809:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895176064/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233849 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233824.689817:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233824.689819:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233824.689820:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895176064:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233824.689823:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895176128:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233824.689833:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233824.689836:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233824/real 1761233824] req@000000006d61dd06 x1846780895176128/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233849 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233824.689860:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895176128/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233849 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233824.689866:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233824.689868:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233824.689869:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895176128:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233824.689885:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895176192:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233824.689899:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233824.689904:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233824/real 1761233824] req@00000000ef655201 x1846780895176192/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233849 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233824.689910:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895176192/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233849 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233824.689915:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233824.689916:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233824.689918:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895176192:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233824.689922:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895176256:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233824.689931:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233824.689935:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233824/real 1761233824] req@000000004130b6bd x1846780895176256/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233849 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233824.689941:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895176256/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233849 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233824.689945:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233824.689946:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233824.689948:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895176256:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233825.457354:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233826.481444:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233827.505365:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233828.529424:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233829.553394:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233829.809296:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233829.809306:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233829.809309:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233829.809319:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6390 00000100:00080000:0.0:1761233829.809323:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233829.809329:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233829.809362:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895176320/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233829.809384:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233829.809387:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233829.809388:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233829.809391:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6390 00000100:00080000:0.0:1761233829.809392:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233829.809395:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233829.809402:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233829.809411:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895176384/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233829.809417:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233829.809420:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895176320:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233829.809421:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233829.809423:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233829.809428:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6390 00000100:00080000:0.0:1761233829.809430:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233829.809432:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233829.809436:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233829.809445:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895176448/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233829.809449:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233829.809452:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233829.809453:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233829.809455:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6390 00000100:00080000:0.0:1761233829.809456:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233829.809458:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233829.809461:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233829.809467:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895176512/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233829.809475:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233829.809478:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233829.809479:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233829.809481:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6390 00000100:00080000:0.0:1761233829.809482:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233829.809484:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233829.809487:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233829.809489:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233829.809496:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895176576/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233829.809503:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233829.809505:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233829.809506:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233829.809509:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6390 00000100:00080000:0.0:1761233829.809512:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233829.809512:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233829/real 1761233829] req@000000009d1c7f73 x1846780895176320/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233854 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233829.809517:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233829.809522:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233829.809532:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895176640/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233829.809536:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895176320/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233854 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233829.809547:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233829.809551:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233829.809552:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233829.809553:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233829.809557:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6390 00000100:00080000:3.0:1761233829.809559:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233829.809560:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233829.809564:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895176320:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233829.809565:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233829.809571:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233829.809579:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895176704/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233829.809591:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895176384:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233829.809611:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233829.809626:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233829/real 1761233829] req@00000000d67bb589 x1846780895176384/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233854 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233829.809634:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895176384/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233854 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233829.809639:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233829.809640:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233829.809642:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895176384:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233829.809645:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895176448:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233829.809656:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233829.809659:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233829/real 1761233829] req@00000000a4a2c44a x1846780895176448/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233854 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233829.809665:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895176448/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233854 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233829.809670:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233829.809671:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233829.809673:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895176448:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233829.809676:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895176512:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233829.809685:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233829.809688:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233829/real 1761233829] req@000000004a74da19 x1846780895176512/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233854 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233829.809694:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895176512/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233854 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233829.809699:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233829.809701:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233829.809702:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895176512:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233829.809705:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895176576:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233829.809715:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233829.809718:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233829/real 1761233829] req@000000002d1b8ef7 x1846780895176576/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233854 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233829.809724:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895176576/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233854 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233829.809728:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233829.809729:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233829.809731:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895176576:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233829.809734:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895176640:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233829.809742:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233829.809746:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233829/real 1761233829] req@000000007bd1ec64 x1846780895176640/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233854 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233829.809751:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895176640/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233854 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233829.809756:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233829.809757:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233829.809758:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895176640:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233829.809781:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895176704:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233829.809806:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233829.809811:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233829/real 1761233829] req@000000001db96463 x1846780895176704/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233854 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233829.809817:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895176704/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233854 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233829.809822:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233829.809823:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233829.809825:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895176704:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233830.577456:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233831.601399:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233832.625384:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233833.649394:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233834.673389:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233834.929318:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233834.929328:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233834.929331:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233834.929341:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6395 00000100:00080000:0.0:1761233834.929344:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233834.929350:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233834.929384:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895176768/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233834.929403:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233834.929406:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233834.929407:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233834.929409:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6395 00000100:00080000:0.0:1761233834.929410:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233834.929412:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233834.929419:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233834.929427:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895176832/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233834.929432:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233834.929434:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233834.929435:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233834.929439:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6395 00000100:00080000:0.0:1761233834.929440:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233834.929442:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233834.929444:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233834.929448:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895176768:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761233834.929452:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895176896/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233834.929461:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233834.929463:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233834.929464:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233834.929466:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6395 00000100:00080000:0.0:1761233834.929467:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233834.929470:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233834.929472:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233834.929481:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895176960/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233834.929489:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233834.929492:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233834.929494:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233834.929497:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6395 00000100:00080000:0.0:1761233834.929499:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233834.929504:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233834.929508:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233834.929519:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895177024/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233834.929530:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233834.929539:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233834.929542:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233834.929543:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233834.929547:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6395 00000100:00080000:0.0:1761233834.929548:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233834.929551:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233834.929556:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233834.929561:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233834/real 1761233834] req@000000005d83493f x1846780895176768/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233859 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233834.929566:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895177088/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233834.929591:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233834.929593:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233834.929594:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233834.929597:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6395 00000100:00080000:0.0:1761233834.929598:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233834.929602:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233834.929602:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005d83493f x1846780895176768/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233859 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:0.0:1761233834.929608:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233834.929616:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895177152/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233834.929621:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233834.929626:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233834.929632:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895176768:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233834.929641:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895176832:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233834.929668:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233834.929678:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233834/real 1761233834] req@00000000f3ed2588 x1846780895176832/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233859 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233834.929706:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f3ed2588 x1846780895176832/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233859 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233834.929714:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233834.929716:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233834.929720:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895176832:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233834.929746:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895176896:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233834.929771:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233834.929781:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233834/real 1761233834] req@000000001db96463 x1846780895176896/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233859 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233834.929794:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895176896/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233859 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233834.929801:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233834.929804:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233834.929807:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895176896:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233834.929813:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895176960:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233834.929834:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233834.929843:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233834/real 1761233834] req@000000007bd1ec64 x1846780895176960/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233859 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233834.929855:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895176960/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233859 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233834.929864:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233834.929866:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233834.929870:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895176960:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233834.929876:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895177024:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233834.929897:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233834.929905:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233834/real 1761233834] req@000000002d1b8ef7 x1846780895177024/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233859 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233834.929916:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895177024/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233859 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233834.929923:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233834.929926:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233834.929929:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895177024:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233834.929935:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895177088:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233834.929955:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233834.929964:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233834/real 1761233834] req@000000004a74da19 x1846780895177088/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233859 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233834.929976:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895177088/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233859 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233834.929985:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233834.929987:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233834.929990:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895177088:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233834.929997:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895177152:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233834.930017:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233834.930025:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233834/real 1761233834] req@00000000a4a2c44a x1846780895177152/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233859 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233834.930038:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895177152/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233859 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233834.930044:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233834.930046:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233834.930049:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895177152:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233835.697429:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233836.721356:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233837.745361:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233838.769372:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233839.793430:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233840.049359:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233840.049369:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233840.049372:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233840.049382:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6400 00000100:00080000:0.0:1761233840.049386:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233840.049393:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233840.049427:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895177216/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233840.049454:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233840.049458:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233840.049460:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233840.049464:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6400 00000100:00080000:0.0:1761233840.049467:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233840.049472:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233840.049482:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233840.049487:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895177216:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:0.0:1761233840.049498:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895177280/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233840.049508:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233840.049511:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233840.049512:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233840.049515:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6400 00000100:00080000:0.0:1761233840.049518:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233840.049521:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233840.049527:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233840.049539:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895177344/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233840.049549:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233840.049552:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233840.049554:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233840.049558:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6400 00000100:00080000:0.0:1761233840.049561:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1761233840.049562:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233840.049568:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233840.049575:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233840.049586:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233840/real 1761233840] req@00000000d67bb589 x1846780895177216/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233865 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233840.049590:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895177408/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233840.049622:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233840.049626:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895177216/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233865 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233840.049629:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233840.049631:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233840.049639:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233840.049640:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6400 00000100:00080000:3.0:1761233840.049643:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233840.049645:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233840.049649:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895177216:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233840.049653:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233840.049660:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233840.049671:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895177280:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761233840.049674:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895177472/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233840.049691:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233840.049694:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233840.049696:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233840.049698:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6400 00000100:00080000:0.0:1761233840.049700:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1761233840.049703:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233840.049707:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233840.049712:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233840.049713:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233840/real 1761233840] req@000000009d1c7f73 x1846780895177280/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233865 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233840.049725:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895177536/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233840.049728:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895177280/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233865 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233840.049757:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233840.049757:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233840.049763:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233840.049764:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233840.049765:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233840.049772:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6400 00000100:00100000:3.0:1761233840.049773:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895177280:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:0.0:1761233840.049775:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233840.049778:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233840.049780:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233840.049787:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895177600/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233840.049801:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895177344:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233840.049827:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233840.049833:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233840/real 1761233840] req@000000004130b6bd x1846780895177344/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233865 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233840.049841:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895177344/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233865 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233840.049846:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233840.049848:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233840.049849:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895177344:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233840.049853:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895177408:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233840.049865:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233840.049869:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233840/real 1761233840] req@00000000ef655201 x1846780895177408/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233865 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233840.049875:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895177408/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233865 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233840.049880:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233840.049881:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233840.049883:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895177408:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233840.049896:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895177472:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233840.049907:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233840.049911:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233840/real 1761233840] req@000000006d61dd06 x1846780895177472/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233865 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233840.049917:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895177472/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233865 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233840.049922:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233840.049923:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233840.049924:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895177472:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233840.049928:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895177536:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233840.049937:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233840.049941:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233840/real 1761233840] req@00000000efa29eb9 x1846780895177536/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233865 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233840.049951:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895177536/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233865 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233840.049958:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233840.049960:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233840.049963:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895177536:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233840.049969:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895177600:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233840.049989:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233840.049997:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233840/real 1761233840] req@000000001f9ec359 x1846780895177600/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233865 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233840.050008:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895177600/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233865 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233840.050016:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233840.050017:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233840.050019:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895177600:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233840.817404:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233841.841417:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233842.865351:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233843.889434:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233844.913358:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233845.169289:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233845.169298:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233845.169302:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233845.169312:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6406 00000100:00080000:0.0:1761233845.169316:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233845.169321:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233845.169353:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895177664/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233845.169374:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233845.169376:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233845.169378:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233845.169380:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6406 00000100:00080000:0.0:1761233845.169381:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233845.169384:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233845.169392:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233845.169401:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895177728/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233845.169407:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895177664:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233845.169410:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233845.169414:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233845.169414:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233845.169417:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6406 00000100:00080000:0.0:1761233845.169419:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233845.169420:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233845.169423:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233845.169431:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895177792/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233845.169436:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233845.169438:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233845.169439:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233845.169441:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6406 00000100:00080000:0.0:1761233845.169443:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233845.169445:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233845.169448:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233845.169454:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895177856/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233845.169459:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233845.169461:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233845.169462:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233845.169465:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6406 00000100:00080000:0.0:1761233845.169466:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1761233845.169469:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233845.169472:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233845.169476:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233845.169483:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895177920/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233845.169489:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233845.169491:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233845/real 1761233845] req@000000005593c994 x1846780895177664/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233870 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233845.169493:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233845.169495:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233845.169502:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6406 00000100:00080000:0.0:1761233845.169505:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233845.169509:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233845.169512:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233845.169517:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895177664/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233870 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233845.169536:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895177984/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233845.169548:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233845.169549:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233845.169555:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233845.169555:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233845.169557:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233845.169562:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6406 00000100:00100000:3.0:1761233845.169562:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895177664:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233845.169566:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233845.169571:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233845.169575:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233845.169582:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895178048/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233845.169590:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895177728:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233845.169618:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233845.169625:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233845/real 1761233845] req@00000000c6e06344 x1846780895177728/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233870 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233845.169632:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895177728/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233870 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233845.169637:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233845.169639:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233845.169641:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895177728:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233845.169644:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895177792:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233845.169656:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233845.169660:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233845/real 1761233845] req@000000004862b808 x1846780895177792/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233870 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233845.169666:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895177792/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233870 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233845.169671:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233845.169672:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233845.169674:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895177792:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233845.169677:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895177856:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233845.169687:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233845.169690:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233845/real 1761233845] req@000000006f2efc8b x1846780895177856/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233870 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233845.169696:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895177856/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233870 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233845.169701:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233845.169703:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233845.169704:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895177856:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233845.169707:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895177920:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233845.169717:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233845.169721:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233845/real 1761233845] req@00000000eca9ddc2 x1846780895177920/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233870 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233845.169727:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895177920/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233870 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233845.169731:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233845.169732:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233845.169734:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895177920:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233845.169736:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895177984:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233845.169745:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233845.169749:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233845/real 1761233845] req@0000000034d49f59 x1846780895177984/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233870 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233845.169754:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895177984/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233870 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233845.169759:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233845.169760:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233845.169762:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895177984:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233845.169784:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895178048:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233845.169796:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233845.169800:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233845/real 1761233845] req@000000005beaae31 x1846780895178048/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233870 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233845.169807:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895178048/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233870 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233845.169811:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233845.169813:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233845.169814:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895178048:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233845.937402:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233846.961367:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233847.985456:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233849.009428:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233850.033378:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233850.289287:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233850.289297:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233850.289301:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233850.289313:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6411 00000100:00080000:2.0:1761233850.289317:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233850.289325:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233850.289372:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895178112/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233850.289395:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233850.289398:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233850.289399:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233850.289403:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6411 00000100:00080000:2.0:1761233850.289405:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233850.289409:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233850.289429:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233850.289453:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895178176/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233850.289453:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895178112:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761233850.289463:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233850.289466:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233850.289467:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233850.289470:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6411 00000100:00080000:2.0:1761233850.289472:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233850.289475:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233850.289480:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233850.289488:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895178240/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233850.289494:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233850.289497:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233850.289498:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233850.289502:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6411 00000100:00080000:2.0:1761233850.289504:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233850.289507:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233850.289510:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233850.289517:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895178304/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233850.289518:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233850.289524:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233850.289526:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233850.289528:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233850.289531:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6411 00000100:00080000:2.0:1761233850.289533:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233850.289536:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233850.289541:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233850.289542:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233850/real 1761233850] req@00000000fd3935a6 x1846780895178112/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233875 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233850.289549:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895178368/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233850.289555:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233850.289558:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1761233850.289559:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895178112/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233875 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233850.289560:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233850.289563:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6411 00000100:00080000:2.0:1761233850.289565:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233850.289568:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233850.289569:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233850.289572:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761233850.289572:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233850.289575:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895178112:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761233850.289579:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895178432/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233850.289590:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233850.289593:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233850.289594:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233850.289596:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895178176:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:2.0:1761233850.289597:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6411 00000100:00080000:2.0:1761233850.289599:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233850.289602:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233850.289605:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233850.289612:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895178496/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233850.289618:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233850.289625:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233850/real 1761233850] req@00000000135e9a6e x1846780895178176/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233875 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233850.289636:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895178176/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233875 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233850.289644:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233850.289646:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233850.289649:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895178176:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233850.289655:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895178240:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233850.289674:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233850.289680:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233850/real 1761233850] req@000000006ff1fdbc x1846780895178240/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233875 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233850.289687:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895178240/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233875 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233850.289692:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233850.289693:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233850.289695:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895178240:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233850.289699:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895178304:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233850.289709:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233850.289712:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233850/real 1761233850] req@0000000088939481 x1846780895178304/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233875 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233850.289718:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895178304/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233875 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233850.289723:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233850.289725:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233850.289726:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895178304:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233850.289729:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895178368:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233850.289739:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233850.289742:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233850/real 1761233850] req@000000006d8c3eb0 x1846780895178368/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233875 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233850.289748:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895178368/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233875 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233850.289753:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233850.289754:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233850.289755:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895178368:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233850.289758:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895178432:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233850.289767:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233850.289770:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233850/real 1761233850] req@000000006c1041ba x1846780895178432/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233875 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233850.289786:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895178432/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233875 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233850.289790:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233850.289792:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233850.289793:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895178432:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233850.289809:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895178496:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233850.289825:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233850.289830:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233850/real 1761233850] req@00000000fd4aa3a2 x1846780895178496/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233875 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233850.289836:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895178496/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233875 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233850.289841:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233850.289842:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233850.289843:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895178496:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233851.057345:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233852.081384:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233853.105470:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233854.129443:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233855.153387:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233855.409411:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233855.409420:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233855.409423:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233855.409432:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6416 00000100:00080000:0.0:1761233855.409435:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233855.409441:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233855.409473:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000078e7052d x1846780895178560/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233855.409497:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233855.409499:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233855.409500:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233855.409503:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6416 00000100:00080000:0.0:1761233855.409505:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233855.409507:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233855.409515:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233855.409524:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000addc3db8 x1846780895178624/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233855.409531:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233855.409533:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233855.409535:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233855.409537:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6416 00000100:00080000:0.0:1761233855.409538:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233855.409542:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895178560:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233855.409543:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233855.409550:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233855.409557:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895178688/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233855.409562:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233855.409565:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233855.409565:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233855.409568:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6416 00000100:00080000:0.0:1761233855.409569:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233855.409571:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233855.409574:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233855.409580:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895178752/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233855.409585:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233855.409587:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233855.409588:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233855.409590:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6416 00000100:00080000:0.0:1761233855.409591:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233855.409593:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233855.409596:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233855.409602:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895178816/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233855.409606:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233855.409608:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233855.409610:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233855.409612:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6416 00000100:00080000:0.0:1761233855.409614:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233855.409616:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233855.409619:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233855.409621:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233855.409629:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895178880/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233855.409635:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233855.409637:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233855.409638:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233855.409641:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6416 00000100:00080000:0.0:1761233855.409642:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233855.409644:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233855/real 1761233855] req@0000000078e7052d x1846780895178560/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233880 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233855.409647:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233855.409652:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233855.409661:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895178944/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233855.409667:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000078e7052d x1846780895178560/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233880 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233855.409676:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233855.409678:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233855.409682:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895178560:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233855.409703:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895178624:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233855.409720:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233855.409725:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233855/real 1761233855] req@00000000addc3db8 x1846780895178624/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233880 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233855.409735:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000addc3db8 x1846780895178624/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233880 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233855.409742:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233855.409744:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233855.409747:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895178624:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233855.409753:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895178688:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233855.409774:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233855.409782:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233855/real 1761233855] req@000000005beaae31 x1846780895178688/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233880 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233855.409795:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895178688/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233880 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233855.409803:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233855.409806:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233855.409809:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895178688:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233855.409815:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895178752:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233855.409837:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233855.409845:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233855/real 1761233855] req@0000000034d49f59 x1846780895178752/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233880 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233855.409856:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895178752/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233880 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233855.409864:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233855.409866:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233855.409869:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895178752:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233855.409875:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895178816:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233855.409892:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233855.409898:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233855/real 1761233855] req@00000000eca9ddc2 x1846780895178816/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233880 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233855.409904:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895178816/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233880 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233855.409909:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233855.409910:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233855.409912:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895178816:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233855.409916:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895178880:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233855.409931:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233855.409938:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233855/real 1761233855] req@000000006f2efc8b x1846780895178880/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233880 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233855.409948:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895178880/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233880 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233855.409955:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233855.409957:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233855.409960:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895178880:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233855.409966:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895178944:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233855.409987:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233855.409995:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233855/real 1761233855] req@000000004862b808 x1846780895178944/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233880 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233855.410006:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895178944/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233880 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233855.410013:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233855.410015:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233855.410018:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895178944:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233856.177415:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233857.201430:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233858.225402:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233859.249413:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233860.273402:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233860.529351:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233860.529360:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233860.529363:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233860.529373:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6421 00000100:00080000:1.0:1761233860.529377:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233860.529382:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233860.529413:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cbad49de x1846780895179008/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233860.529435:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233860.529438:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233860.529439:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233860.529441:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6421 00000100:00080000:1.0:1761233860.529443:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233860.529445:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233860.529452:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233860.529460:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e467e107 x1846780895179072/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233860.529466:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233860.529469:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233860.529470:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233860.529473:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6421 00000100:00080000:1.0:1761233860.529474:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233860.529475:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233860.529479:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233860.529480:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbad49de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895179008:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233860.529487:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090800dde x1846780895179136/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233860.529493:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233860.529495:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233860.529497:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233860.529500:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6421 00000100:00080000:1.0:1761233860.529502:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233860.529504:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233860.529507:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233860.529516:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000328368c8 x1846780895179200/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233860.529522:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233860.529524:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233860.529525:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233860.529527:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6421 00000100:00080000:1.0:1761233860.529528:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233860.529531:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233860.529534:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233860.529540:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a611ee34 x1846780895179264/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233860.529546:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:1.0:1761233860.529550:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233860.529552:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233860.529554:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233860.529558:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6421 00000100:00080000:1.0:1761233860.529560:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233860.529563:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233860.529566:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233860.529569:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233860/real 1761233860] req@00000000cbad49de x1846780895179008/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233885 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233860.529574:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001aedf9b1 x1846780895179328/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233860.529588:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233860.529590:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cbad49de x1846780895179008/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233885 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233860.529594:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233860.529597:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233860.529602:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233860.529604:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6421 00000100:00080000:3.0:1761233860.529605:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:1.0:1761233860.529609:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233860.529611:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbad49de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895179008:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233860.529620:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233860.529620:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e467e107 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895179072:192.168.201.157@tcp:38:kworker/u8:3.0 00000008:00100000:1.0:1761233860.529629:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233860.529647:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d74b2e3c x1846780895179392/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233860.529648:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233860.529656:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233860/real 1761233860] req@00000000e467e107 x1846780895179072/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233885 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233860.529669:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e467e107 x1846780895179072/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233885 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233860.529678:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233860.529681:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233860.529685:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e467e107 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895179072:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233860.529713:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090800dde pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895179136:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233860.529738:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233860.529747:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233860/real 1761233860] req@0000000090800dde x1846780895179136/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233885 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233860.529759:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090800dde x1846780895179136/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233885 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233860.529768:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233860.529770:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233860.529773:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090800dde pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895179136:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233860.529779:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895179200:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233860.529799:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233860.529808:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233860/real 1761233860] req@00000000328368c8 x1846780895179200/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233885 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233860.529820:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000328368c8 x1846780895179200/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233885 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233860.529828:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233860.529830:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233860.529833:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895179200:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233860.529839:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895179264:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233860.529859:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233860.529879:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233860/real 1761233860] req@00000000a611ee34 x1846780895179264/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233885 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233860.529895:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a611ee34 x1846780895179264/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233885 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233860.529903:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233860.529906:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233860.529908:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895179264:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233860.529914:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895179328:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233860.529936:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233860.529945:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233860/real 1761233860] req@000000001aedf9b1 x1846780895179328/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233885 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233860.529956:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001aedf9b1 x1846780895179328/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233885 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233860.529963:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233860.529966:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233860.529969:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895179328:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233860.529974:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895179392:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233860.529995:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233860.530003:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233860/real 1761233860] req@00000000d74b2e3c x1846780895179392/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233885 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233860.530014:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d74b2e3c x1846780895179392/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233885 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233860.530021:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233860.530024:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233860.530026:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895179392:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233861.297437:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233862.321416:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233863.345410:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233864.369389:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233865.393377:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233865.649360:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233865.649372:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233865.649377:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233865.649390:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6426 00000100:00080000:1.0:1761233865.649394:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233865.649401:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233865.649448:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ba3340f x1846780895179456/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233865.649473:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233865.649476:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233865.649477:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233865.649480:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6426 00000100:00080000:1.0:1761233865.649481:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233865.649483:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233865.649493:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233865.649502:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c652d0ad x1846780895179520/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233865.649508:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233865.649514:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895179456:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233865.649515:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233865.649518:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233865.649523:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6426 00000100:00080000:1.0:1761233865.649525:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233865.649527:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233865.649530:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233865.649538:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000568f2a96 x1846780895179584/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233865.649542:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233865.649545:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233865.649546:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233865.649548:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6426 00000100:00080000:1.0:1761233865.649550:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233865.649552:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233865.649555:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233865.649562:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000038209f5 x1846780895179648/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233865.649567:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233865.649569:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233865.649570:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233865.649573:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6426 00000100:00080000:1.0:1761233865.649574:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233865.649577:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233865.649579:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233865.649586:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000c736bb2 x1846780895179712/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233865.649591:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233865.649593:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233865.649595:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233865.649599:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6426 00000800:00000100:3.0:1761233865.649600:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:1.0:1761233865.649601:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233865.649605:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233865.649607:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233865.649614:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a95774fb x1846780895179776/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233865.649620:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233865.649622:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233865.649623:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233865.649626:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6426 00000100:00080000:1.0:1761233865.649629:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233865.649629:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233865/real 1761233865] req@000000007ba3340f x1846780895179456/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233890 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233865.649635:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233865.649640:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233865.649649:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cb5be845 x1846780895179840/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233865.649653:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007ba3340f x1846780895179456/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233890 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233865.649672:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233865.649675:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233865.649679:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895179456:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233865.649705:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895179520:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233865.649721:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233865.649726:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233865/real 1761233865] req@00000000c652d0ad x1846780895179520/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233890 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233865.649733:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c652d0ad x1846780895179520/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233890 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233865.649738:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233865.649740:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233865.649742:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895179520:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233865.649745:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895179584:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233865.649766:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233865.649770:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233865/real 1761233865] req@00000000568f2a96 x1846780895179584/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233890 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233865.649776:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000568f2a96 x1846780895179584/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233890 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233865.649781:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233865.649782:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233865.649784:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895179584:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233865.649787:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895179648:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233865.649797:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233865.649800:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233865/real 1761233865] req@00000000038209f5 x1846780895179648/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233890 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233865.649806:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000038209f5 x1846780895179648/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233890 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233865.649811:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233865.649813:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233865.649815:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895179648:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233865.649817:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895179712:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233865.649827:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233865.649831:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233865/real 1761233865] req@000000000c736bb2 x1846780895179712/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233890 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233865.649836:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000000c736bb2 x1846780895179712/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233890 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233865.649841:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233865.649842:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233865.649844:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895179712:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233865.649847:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895179776:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233865.649857:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233865.649860:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233865/real 1761233865] req@00000000a95774fb x1846780895179776/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233890 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233865.649866:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a95774fb x1846780895179776/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233890 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233865.649870:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233865.649872:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233865.649873:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895179776:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233865.649877:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895179840:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233865.649887:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233865.649891:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233865/real 1761233865] req@00000000cb5be845 x1846780895179840/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233890 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233865.649897:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cb5be845 x1846780895179840/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233890 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233865.649902:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233865.649903:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233865.649905:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895179840:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233866.417345:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233867.441421:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233868.465338:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233869.489421:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233870.513409:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233870.769300:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233870.769310:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233870.769313:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233870.769324:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6431 00000100:00080000:1.0:1761233870.769328:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233870.769336:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233870.769370:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000880f16b4 x1846780895179904/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233870.769391:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233870.769394:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233870.769395:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233870.769398:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6431 00000100:00080000:1.0:1761233870.769399:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233870.769401:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233870.769408:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233870.769416:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000053e52331 x1846780895179968/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233870.769423:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233870.769426:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1761233870.769427:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895179904:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233870.769429:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233870.769435:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6431 00000100:00080000:1.0:1761233870.769437:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233870.769439:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233870.769442:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233870.769451:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090ac513e x1846780895180032/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233870.769456:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233870.769457:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233870.769459:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233870.769461:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6431 00000100:00080000:1.0:1761233870.769463:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233870.769465:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233870.769467:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233870.769473:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d66930c2 x1846780895180096/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233870.769478:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233870.769482:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233870.769484:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233870.769487:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6431 00000100:00080000:1.0:1761233870.769489:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1761233870.769494:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:1.0:1761233870.769496:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233870.769503:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233870.769515:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ffe86d04 x1846780895180160/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233870.769517:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233870/real 1761233870] req@00000000880f16b4 x1846780895179904/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233895 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233870.769545:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233870.769549:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000880f16b4 x1846780895179904/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233895 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233870.769550:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233870.769553:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233870.769564:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6431 00000100:00080000:3.0:1761233870.769566:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233870.769568:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233870.769570:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:1.0:1761233870.769579:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233870.769579:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895179904:192.168.201.157@tcp:250:kworker/u8:3.0 00000008:00100000:1.0:1761233870.769590:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233870.769602:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df5b2270 x1846780895180224/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233870.769608:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895179968:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:1.0:1761233870.769620:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233870.769625:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233870.769627:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233870.769633:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6431 00000100:00080000:1.0:1761233870.769635:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233870.769640:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233870.769646:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233870.769647:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233870.769660:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233870/real 1761233870] req@0000000053e52331 x1846780895179968/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233895 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233870.769663:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df0b94a8 x1846780895180288/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233870.769692:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000053e52331 x1846780895179968/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233895 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233870.769698:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233870.769700:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233870.769703:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895179968:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233870.769709:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895180032:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233870.769726:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233870.769731:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233870/real 1761233870] req@0000000090ac513e x1846780895180032/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233895 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233870.769737:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090ac513e x1846780895180032/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233895 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233870.769742:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233870.769743:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233870.769744:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895180032:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233870.769748:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895180096:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233870.769757:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233870.769761:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233870/real 1761233870] req@00000000d66930c2 x1846780895180096/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233895 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233870.769767:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d66930c2 x1846780895180096/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233895 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233870.769772:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233870.769773:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233870.769775:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895180096:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233870.769778:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895180160:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233870.769787:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233870.769791:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233870/real 1761233870] req@00000000ffe86d04 x1846780895180160/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233895 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233870.769796:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ffe86d04 x1846780895180160/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233895 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233870.769800:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233870.769802:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233870.769803:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895180160:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233870.769817:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895180224:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233870.769827:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233870.769831:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233870/real 1761233870] req@00000000df5b2270 x1846780895180224/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233895 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233870.769837:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df5b2270 x1846780895180224/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233895 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233870.769841:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233870.769842:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233870.769844:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895180224:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233870.769847:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895180288:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233870.769857:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233870.769861:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233870/real 1761233870] req@00000000df0b94a8 x1846780895180288/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233895 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233870.769867:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df0b94a8 x1846780895180288/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233895 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233870.769871:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233870.769872:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233870.769873:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895180288:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233871.537414:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233872.561447:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233873.585402:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233874.609434:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233875.633418:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233875.889298:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233875.889308:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233875.889311:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233875.889321:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6436 00000100:00080000:1.0:1761233875.889324:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233875.889330:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233875.889359:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004361f727 x1846780895180352/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233875.889378:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233875.889381:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233875.889382:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233875.889384:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6436 00000100:00080000:1.0:1761233875.889386:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233875.889388:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233875.889395:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233875.889403:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006a5e9865 x1846780895180416/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233875.889409:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233875.889411:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233875.889412:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233875.889415:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6436 00000100:00080000:1.0:1761233875.889416:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233875.889417:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233875.889420:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233875.889424:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004361f727 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895180352:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233875.889429:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac65ecb7 x1846780895180480/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233875.889436:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233875.889438:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233875.889439:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233875.889441:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6436 00000100:00080000:1.0:1761233875.889442:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233875.889444:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233875.889447:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233875.889456:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df0b94a8 x1846780895180544/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233875.889461:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233875.889463:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233875.889464:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233875.889466:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6436 00000100:00080000:1.0:1761233875.889468:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233875.889470:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233875.889473:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233875.889479:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df5b2270 x1846780895180608/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233875.889483:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233875.889487:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233875.889488:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233875.889492:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6436 00000100:00080000:1.0:1761233875.889493:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233875.889497:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233875.889501:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233875.889512:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ffe86d04 x1846780895180672/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233875.889521:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233875.889525:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a5e9865 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895180416:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:1.0:1761233875.889526:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233875.889528:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233875.889537:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6436 00000100:00080000:1.0:1761233875.889540:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233875.889544:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233875.889550:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233875.889563:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d66930c2 x1846780895180736/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233875.889573:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac65ecb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895180480:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233875.889607:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895180544:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233875.889627:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895180608:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233875.889650:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895180672:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233875.889678:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895180736:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233876.657464:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233877.681424:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233878.705404:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1761233878.961455:0:2208:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:2.0:1761233878.961492:0:2208:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1761233878.961505:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761233878.961536:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761233878.961545:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761233878.961554:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761233878.961562:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:3.0:1761233878.961568:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233875/real 1761233878] req@00000000ac65ecb7 x1846780895180480/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233900 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000800:00000100:2.0:1761233878.961570:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761233878.961578:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:3.0:1761233878.961591:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac65ecb7 x1846780895180480/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233900 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233878.961603:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233878.961607:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233878.961611:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac65ecb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895180480:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233878.961617:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233875/real 1761233878] req@00000000df0b94a8 x1846780895180544/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233900 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233878.961628:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df0b94a8 x1846780895180544/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233900 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233878.961634:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233878.961637:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233878.961639:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895180544:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233878.961644:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233875/real 1761233878] req@00000000df5b2270 x1846780895180608/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233900 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233878.961654:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df5b2270 x1846780895180608/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233900 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233878.961660:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233878.961663:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233878.961666:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895180608:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233878.961672:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233875/real 1761233878] req@00000000ffe86d04 x1846780895180672/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233900 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233878.961682:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ffe86d04 x1846780895180672/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233900 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233878.961688:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233878.961691:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233878.961694:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895180672:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233878.961699:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233875/real 1761233878] req@000000004361f727 x1846780895180352/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233900 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233878.961709:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004361f727 x1846780895180352/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233900 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233878.961715:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233878.961718:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233878.961721:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004361f727 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895180352:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233878.961726:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233875/real 1761233878] req@000000006a5e9865 x1846780895180416/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233900 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233878.961737:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006a5e9865 x1846780895180416/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233900 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233878.961743:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233878.961746:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233878.961759:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a5e9865 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895180416:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233878.961800:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233875/real 1761233878] req@00000000d66930c2 x1846780895180736/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233900 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233878.961811:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d66930c2 x1846780895180736/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233900 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233878.961818:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233878.961821:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233878.961823:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895180736:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233879.729424:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233880.753357:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233881.009459:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233881.009469:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233881.009472:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233881.009485:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6441 00000100:00080000:2.0:1761233881.009489:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233881.009496:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233881.009532:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895180800/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233881.009561:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233881.009566:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233881.009568:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233881.009572:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6441 00000100:00080000:2.0:1761233881.009574:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233881.009577:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233881.009585:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233881.009597:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895180864/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233881.009609:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233881.009613:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1761233881.009615:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895180800:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761233881.009616:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233881.009625:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6441 00000100:00080000:2.0:1761233881.009627:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233881.009630:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233881.009635:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233881.009647:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895180928/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233881.009656:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233881.009660:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233881.009661:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233881.009664:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6441 00000100:00080000:2.0:1761233881.009668:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233881.009672:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233881.009676:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233881.009687:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895180992/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233881.009698:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1761233881.009699:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233881.009704:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233881.009706:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233881.009712:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6441 00000100:00080000:2.0:1761233881.009714:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233881.009719:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233881.009724:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233881.009728:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233881/real 1761233881] req@00000000f4e415d3 x1846780895180800/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233906 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233881.009739:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895181056/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233881.009753:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233881.009757:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1761233881.009757:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895180800/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233906 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233881.009760:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233881.009770:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6441 00000100:00080000:2.0:1761233881.009775:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233881.009778:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233881.009782:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:3.0:1761233881.009785:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000008:00100000:2.0:1761233881.009790:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233881.009794:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895180800:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761233881.009809:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895181120/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233881.009821:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233881.009825:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233881.009827:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761233881.009830:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895180864:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:2.0:1761233881.009836:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6441 00000100:00080000:2.0:1761233881.009841:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233881.009845:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233881.009851:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233881.009865:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895181184/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233881.009875:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233881.009893:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233881/real 1761233881] req@0000000099b7e5d9 x1846780895180864/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233906 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233881.009909:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895180864/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233906 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233881.009919:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233881.009921:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233881.009925:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895180864:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233881.009932:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895180928:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233881.009955:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233881.010018:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233881/real 1761233881] req@00000000c10cebe6 x1846780895180928/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233906 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233881.010029:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895180928/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233906 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233881.010036:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233881.010038:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233881.010040:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895180928:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233881.010046:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895180992:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233881.010066:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233881.010074:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233881/real 1761233881] req@00000000a1e8c6bb x1846780895180992/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233906 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233881.010096:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895180992/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233906 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233881.010103:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233881.010105:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233881.010108:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895180992:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233881.010114:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895181056:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233881.010134:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233881.010142:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233881/real 1761233881] req@000000008ebccab5 x1846780895181056/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233906 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233881.010152:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895181056/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233906 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233881.010159:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233881.010160:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233881.010163:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895181056:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233881.010169:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895181120:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233881.010186:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233881.010193:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233881/real 1761233881] req@000000004ae5bc9b x1846780895181120/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233906 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233881.010228:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895181120/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233906 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233881.010235:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233881.010237:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233881.010240:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895181120:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233881.010264:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895181184:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233881.010287:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233881.010294:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233881/real 1761233881] req@00000000ac7d8d5b x1846780895181184/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233906 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233881.010304:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895181184/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233906 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233881.010311:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233881.010313:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233881.010315:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895181184:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233881.777376:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233882.801368:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233883.825356:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233884.849409:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233885.873423:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233886.129314:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233886.129325:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233886.129330:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233886.129341:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6447 00000100:00080000:2.0:1761233886.129345:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233886.129352:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233886.129391:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895181248/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233886.129421:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233886.129425:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233886.129426:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233886.129429:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6447 00000100:00080000:2.0:1761233886.129431:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233886.129434:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233886.129442:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233886.129452:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895181312/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233886.129460:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233886.129463:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233886.129465:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233886.129467:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6447 00000100:00080000:2.0:1761233886.129469:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233886.129471:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233886.129474:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233886.129481:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895181376/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233886.129487:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233886.129489:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233886.129490:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233886.129508:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6447 00000100:00080000:2.0:1761233886.129509:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233886.129512:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233886.129515:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233886.129521:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895181440/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233886.129526:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233886.129528:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233886.129529:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233886.129532:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6447 00000100:00080000:2.0:1761233886.129533:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233886.129536:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233886.129539:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233886.129545:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895181248:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761233886.129547:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895181504/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233886.129555:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233886.129557:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233886.129558:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233886.129560:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6447 00000100:00080000:2.0:1761233886.129561:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233886.129563:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233886.129566:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233886.129573:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895181568/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233886.129578:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233886.129580:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233886.129580:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233886.129582:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6447 00000100:00080000:2.0:1761233886.129584:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233886.129585:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233886.129589:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233886.129594:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895181632/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233886.129611:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233886.129630:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233886/real 1761233886] req@00000000ac7d8d5b x1846780895181248/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233911 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233886.129642:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895181248/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233911 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233886.129649:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233886.129651:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233886.129654:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895181248:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233886.129658:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895181312:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233886.129671:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233886.129675:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233886/real 1761233886] req@000000004ae5bc9b x1846780895181312/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233911 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233886.129686:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895181312/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233911 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233886.129691:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233886.129692:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233886.129693:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895181312:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233886.129696:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895181376:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233886.129706:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233886.129710:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233886/real 1761233886] req@000000008ebccab5 x1846780895181376/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233911 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233886.129715:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895181376/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233911 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233886.129719:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233886.129720:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233886.129722:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895181376:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233886.129725:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895181440:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233886.129735:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233886.129738:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233886/real 1761233886] req@00000000a1e8c6bb x1846780895181440/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233911 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233886.129744:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895181440/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233911 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233886.129749:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233886.129750:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233886.129752:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895181440:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233886.129777:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895181504:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233886.129788:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233886.129792:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233886/real 1761233886] req@00000000c10cebe6 x1846780895181504/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233911 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233886.129799:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895181504/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233911 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233886.129803:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233886.129804:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233886.129806:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895181504:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233886.129809:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895181568:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233886.129818:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233886.129824:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233886/real 1761233886] req@0000000099b7e5d9 x1846780895181568/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233911 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233886.129834:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895181568/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233911 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233886.129841:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233886.129843:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233886.129846:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895181568:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233886.129852:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895181632:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233886.129874:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233886.129904:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233886/real 1761233886] req@00000000f4e415d3 x1846780895181632/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233911 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233886.129911:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895181632/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233911 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233886.129916:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233886.129917:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233886.129919:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895181632:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233886.897395:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233887.921338:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233888.945485:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233889.969411:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233890.993446:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233891.249371:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233891.249382:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233891.249387:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233891.249399:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6452 00000100:00080000:2.0:1761233891.249403:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233891.249410:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233891.249450:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895181696/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233891.249479:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233891.249482:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233891.249484:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233891.249486:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6452 00000100:00080000:2.0:1761233891.249488:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233891.249491:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233891.249502:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233891.249513:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895181760/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233891.249521:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233891.249524:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233891.249525:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233891.249530:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6452 00000100:00080000:2.0:1761233891.249531:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233891.249534:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233891.249539:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233891.249548:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895181824/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233891.249550:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895181696:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761233891.249558:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233891.249570:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233891.249572:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233891.249575:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6452 00000100:00080000:2.0:1761233891.249577:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233891.249579:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233891.249582:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233891.249593:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895181888/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233891.249601:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233891.249603:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233891.249605:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233891.249607:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6452 00000100:00080000:2.0:1761233891.249609:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233891.249612:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233891.249615:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233891.249624:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895181952/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233891.249634:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233891.249636:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233891.249637:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233891.249640:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6452 00000100:00080000:2.0:1761233891.249641:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233891.249644:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233891.249649:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233891.249656:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895182016/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233891.249667:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1761233891.249668:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233891.249674:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233891.249677:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233891.249684:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6452 00000100:00080000:2.0:1761233891.249687:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233891.249691:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233891.249696:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233891.249720:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233891/real 1761233891] req@00000000fd4aa3a2 x1846780895181696/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233916 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233891.249727:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895182080/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233891.249829:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895181696/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233916 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233891.249853:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233891.249856:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233891.249861:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895181696:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233891.249868:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895181760:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233891.249890:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233891.249897:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233891/real 1761233891] req@000000006c1041ba x1846780895181760/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233916 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233891.249906:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895181760/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233916 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233891.249911:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233891.249913:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233891.249916:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895181760:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233891.249945:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895181824:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233891.249959:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233891.249963:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233891/real 1761233891] req@000000006d8c3eb0 x1846780895181824/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233916 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233891.249971:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895181824/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233916 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233891.249976:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233891.249978:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233891.249980:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895181824:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233891.249985:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895181888:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233891.249996:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233891.250000:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233891/real 1761233891] req@0000000088939481 x1846780895181888/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233916 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233891.250007:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895181888/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233916 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233891.250018:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233891.250020:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233891.250022:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895181888:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233891.250026:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895181952:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233891.250059:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233891.250067:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233891/real 1761233891] req@000000006ff1fdbc x1846780895181952/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233916 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233891.250078:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895181952/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233916 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233891.250086:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233891.250088:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233891.250091:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895181952:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233891.250097:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895182016:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233891.250114:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233891.250120:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233891/real 1761233891] req@00000000135e9a6e x1846780895182016/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233916 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233891.250140:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895182016/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233916 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233891.250147:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233891.250150:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233891.250152:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895182016:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233891.250159:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895182080:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233891.250176:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233891.250183:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233891/real 1761233891] req@00000000fd3935a6 x1846780895182080/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233916 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233891.250194:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895182080/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233916 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233891.250202:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233891.250204:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233891.250206:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895182080:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233892.017461:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233893.041404:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233894.065439:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233895.089447:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233896.113388:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233896.369355:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233896.369364:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233896.369367:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233896.369377:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6457 00000100:00080000:0.0:1761233896.369380:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233896.369386:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233896.369415:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895182144/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233896.369436:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233896.369439:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233896.369440:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233896.369442:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6457 00000100:00080000:0.0:1761233896.369444:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233896.369446:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233896.369453:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233896.369460:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895182208/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233896.369466:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233896.369468:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233896.369469:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233896.369472:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6457 00000100:00080000:0.0:1761233896.369480:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233896.369483:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233896.369486:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233896.369495:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895182144:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761233896.369498:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895182272/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233896.369506:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233896.369508:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233896.369509:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233896.369511:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6457 00000100:00080000:0.0:1761233896.369513:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233896.369515:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233896.369518:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233896.369525:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895182336/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233896.369531:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233896.369533:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233896.369534:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233896.369536:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6457 00000100:00080000:0.0:1761233896.369538:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233896.369540:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233896.369543:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233896.369548:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895182400/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233896.369553:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233896.369554:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233896.369555:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233896.369557:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6457 00000100:00080000:0.0:1761233896.369558:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1761233896.369561:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233896.369564:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233896.369568:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233896.369577:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895182464/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233896.369582:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233896.369585:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233896/real 1761233896] req@00000000c6e06344 x1846780895182144/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233921 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233896.369586:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233896.369588:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233896.369595:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6457 00000100:00080000:0.0:1761233896.369597:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233896.369600:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233896.369603:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233896.369605:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895182144/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233921 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233896.369611:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895182528/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233896.369616:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233896.369620:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233896.369625:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895182144:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233896.369631:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895182208:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233896.369648:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233896.369653:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233896/real 1761233896] req@000000005593c994 x1846780895182208/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233921 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233896.369660:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895182208/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233921 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233896.369665:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233896.369666:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233896.369668:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895182208:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233896.369688:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895182272:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233896.369699:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233896.369703:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233896/real 1761233896] req@000000001f9ec359 x1846780895182272/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233921 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233896.369710:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895182272/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233921 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233896.369714:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233896.369728:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233896.369732:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895182272:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233896.369736:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895182336:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233896.369749:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233896.369755:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233896/real 1761233896] req@00000000efa29eb9 x1846780895182336/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233921 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233896.369761:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895182336/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233921 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233896.369767:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233896.369768:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233896.369770:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895182336:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233896.369773:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895182400:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233896.369782:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233896.369786:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233896/real 1761233896] req@000000006d61dd06 x1846780895182400/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233921 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233896.369792:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895182400/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233921 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233896.369797:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233896.369798:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233896.369799:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895182400:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233896.369802:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895182464:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233896.369811:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233896.369815:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233896/real 1761233896] req@00000000ef655201 x1846780895182464/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233921 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233896.369821:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895182464/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233921 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233896.369825:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233896.369826:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233896.369828:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895182464:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233896.369832:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895182528:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233896.369841:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233896.369845:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233896/real 1761233896] req@000000004130b6bd x1846780895182528/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233921 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233896.369851:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895182528/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233921 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233896.369855:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233896.369856:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233896.369858:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895182528:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233897.137400:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233898.161409:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233899.185446:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233900.209400:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233901.233351:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233901.489344:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233901.489353:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233901.489357:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233901.489366:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6462 00000100:00080000:2.0:1761233901.489369:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233901.489376:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233901.489406:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895182592/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233901.489426:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233901.489429:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233901.489430:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233901.489433:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6462 00000100:00080000:2.0:1761233901.489434:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233901.489436:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233901.489445:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233901.489454:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895182656/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233901.489461:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233901.489463:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233901.489464:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233901.489469:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6462 00000100:00100000:3.0:1761233901.489470:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895182592:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761233901.489471:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233901.489476:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233901.489480:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233901.489488:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895182720/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233901.489493:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233901.489495:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233901.489496:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233901.489499:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6462 00000100:00080000:2.0:1761233901.489500:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233901.489502:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233901.489504:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233901.489510:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895182784/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233901.489515:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233901.489517:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233901.489517:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233901.489520:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6462 00000100:00080000:2.0:1761233901.489521:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233901.489524:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233901.489526:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233901.489532:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895182848/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233901.489537:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233901.489542:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233901.489544:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233901.489546:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233901.489550:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6462 00000100:00080000:2.0:1761233901.489551:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233901.489553:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233901.489556:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233901.489562:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233901/real 1761233901] req@000000007091f547 x1846780895182592/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233926 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233901.489564:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895182912/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233901.489584:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233901.489586:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233901.489587:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233901.489592:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6462 00000100:00100000:3.0:1761233901.489593:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895182592/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233926 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233901.489596:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233901.489600:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233901.489604:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761233901.489609:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1761233901.489612:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895182976/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233901.489612:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233901.489619:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895182592:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761233901.489671:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895182656:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233901.489690:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233901.489696:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233901/real 1761233901] req@0000000064a2cea0 x1846780895182656/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233926 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233901.489703:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895182656/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233926 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233901.489708:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233901.489710:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233901.489712:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895182656:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233901.489715:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895182720:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233901.489726:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233901.489729:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233901/real 1761233901] req@0000000033765045 x1846780895182720/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233926 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233901.489735:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895182720/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233926 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233901.489740:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233901.489741:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233901.489742:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895182720:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233901.489745:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895182784:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233901.489754:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233901.489758:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233901/real 1761233901] req@000000009d315fa5 x1846780895182784/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233926 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233901.489764:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895182784/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233926 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233901.489768:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233901.489770:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233901.489772:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895182784:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233901.489775:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895182848:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233901.489785:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233901.489789:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233901/real 1761233901] req@0000000040d5f361 x1846780895182848/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233926 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233901.489795:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895182848/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233926 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233901.489799:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233901.489800:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233901.489802:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895182848:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233901.489805:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895182912:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233901.489814:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233901.489818:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233901/real 1761233901] req@000000007dcf84ae x1846780895182912/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233926 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233901.489824:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895182912/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233926 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233901.489828:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233901.489829:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233901.489831:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895182912:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233901.489834:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895182976:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233901.489844:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233901.489848:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233901/real 1761233901] req@000000004f29ca26 x1846780895182976/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233926 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233901.489854:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895182976/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233926 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233901.489859:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233901.489860:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233901.489861:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895182976:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233902.257376:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233903.281345:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233904.305383:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233905.329342:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233906.353390:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233906.609346:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233906.609356:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233906.609360:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233906.609371:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6467 00000100:00080000:0.0:1761233906.609375:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233906.609382:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233906.609415:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895183040/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233906.609445:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233906.609449:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233906.609450:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233906.609455:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6467 00000100:00080000:0.0:1761233906.609457:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233906.609461:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233906.609469:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233906.609480:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895183104/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233906.609490:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233906.609494:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233906.609496:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233906.609500:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6467 00000100:00080000:0.0:1761233906.609503:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233906.609508:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233906.609509:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895183040:192.168.201.157@tcp:250:kworker/u8:2.0 00000008:00100000:0.0:1761233906.609513:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233906.609524:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895183168/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233906.609533:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233906.609537:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233906.609538:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233906.609541:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6467 00000100:00080000:0.0:1761233906.609544:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233906.609547:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233906.609552:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233906.609578:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895183232/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233906.609587:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233906.609590:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233906.609592:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233906.609595:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6467 00000100:00080000:0.0:1761233906.609598:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1761233906.609600:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233906.609604:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233906.609610:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233906.609619:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895183296/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233906.609628:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233906/real 1761233906] req@000000009d1c7f73 x1846780895183040/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233931 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233906.609632:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233906.609639:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233906.609642:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233906.609649:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6467 00000100:00080000:0.0:1761233906.609651:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233906.609655:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233906.609658:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895183040/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233931 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000008:00100000:0.0:1761233906.609660:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233906.609672:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895183360/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233906.609673:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233906.609679:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233906.609685:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233906.609688:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895183040:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233906.609690:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233906.609692:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233906.609700:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6467 00000100:00080000:0.0:1761233906.609702:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233906.609703:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895183104:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:0.0:1761233906.609709:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233906.609715:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233906.609728:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895183424/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233906.609738:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233906.609750:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233906/real 1761233906] req@00000000d67bb589 x1846780895183104/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233931 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233906.609771:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895183104/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233931 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233906.609779:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233906.609782:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233906.609785:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895183104:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233906.609811:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895183168:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233906.609834:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233906.609844:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233906/real 1761233906] req@00000000a4a2c44a x1846780895183168/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233931 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233906.609855:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895183168/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233931 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233906.609863:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233906.609865:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233906.609868:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895183168:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233906.609874:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895183232:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233906.609892:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233906.609900:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233906/real 1761233906] req@000000004a74da19 x1846780895183232/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233931 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233906.609911:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895183232/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233931 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233906.609918:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233906.609920:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233906.609923:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895183232:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233906.609929:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895183296:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761233906.609948:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233906.609957:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233906/real 1761233906] req@000000002d1b8ef7 x1846780895183296/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233931 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233906.609969:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895183296/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233931 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233906.609977:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233906.609979:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233906.609982:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895183296:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761233906.609987:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895183360:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233906.610007:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233906.610015:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233906/real 1761233906] req@000000007bd1ec64 x1846780895183360/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233931 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233906.610026:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895183360/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233931 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233906.610034:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233906.610036:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233906.610039:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895183360:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761233906.610044:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895183424:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761233906.610062:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233906.610071:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233906/real 1761233906] req@000000001db96463 x1846780895183424/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233931 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761233906.610081:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895183424/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233931 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761233906.610088:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233906.610091:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233906.610093:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895183424:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233907.377388:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233908.401427:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233909.425428:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233910.449437:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233911.473385:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233911.729311:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233911.729321:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233911.729325:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233911.729335:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6472 00000100:00080000:0.0:1761233911.729338:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233911.729345:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233911.729376:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895183488/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233911.729400:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233911.729403:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233911.729404:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233911.729407:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6472 00000100:00080000:0.0:1761233911.729408:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233911.729411:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233911.729418:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233911.729426:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895183552/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233911.729432:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895183488:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233911.729436:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233911.729439:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233911.729440:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233911.729444:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6472 00000100:00080000:0.0:1761233911.729446:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233911.729448:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233911.729452:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233911.729461:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895183616/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233911.729466:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233911.729469:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233911.729470:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233911.729472:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6472 00000100:00080000:0.0:1761233911.729500:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233911.729503:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233911.729506:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233911.729516:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895183680/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761233911.729519:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233911.729525:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233911.729528:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233911.729529:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233911.729532:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6472 00000100:00080000:0.0:1761233911.729534:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233911.729536:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233911.729538:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233911.729543:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233911/real 1761233911] req@00000000f3ed2588 x1846780895183488/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233936 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233911.729546:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895183744/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233911.729566:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233911.729568:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233911.729570:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761233911.729573:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f3ed2588 x1846780895183488/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233936 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233911.729576:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6472 00000100:00080000:0.0:1761233911.729581:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233911.729586:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:1.0:1761233911.729587:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1761233911.729589:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1761233911.729591:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233911.729596:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895183488:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761233911.729601:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895183808/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233911.729618:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233911.729620:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895183552:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:0.0:1761233911.729625:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233911.729628:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233911.729633:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6472 00000100:00080000:0.0:1761233911.729635:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233911.729639:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:1.0:1761233911.729643:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:0.0:1761233911.729648:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233911.729651:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233911/real 1761233911] req@000000005d83493f x1846780895183552/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233936 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233911.729659:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005d83493f x1846780895183552/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233936 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233911.729665:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895183872/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233911.729681:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233911.729685:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233911.729690:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895183552:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233911.729697:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895183616:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233911.729719:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233911.729728:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233911/real 1761233911] req@000000001db96463 x1846780895183616/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233936 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233911.729738:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895183616/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233936 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233911.729746:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233911.729748:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233911.729751:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895183616:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233911.729757:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895183680:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233911.729777:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233911.729784:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233911/real 1761233911] req@000000007bd1ec64 x1846780895183680/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233936 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233911.729797:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895183680/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233936 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233911.729806:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233911.729808:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233911.729810:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895183680:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233911.729815:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895183744:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233911.729829:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233911.729834:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233911/real 1761233911] req@000000002d1b8ef7 x1846780895183744/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233936 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233911.729841:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895183744/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233936 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233911.729845:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233911.729847:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233911.729848:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895183744:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233911.729862:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895183808:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233911.729873:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233911.729878:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233911/real 1761233911] req@000000004a74da19 x1846780895183808/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233936 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233911.729885:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895183808/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233936 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233911.729889:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233911.729890:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233911.729892:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895183808:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233911.729895:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895183872:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233911.729903:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233911.729907:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233911/real 1761233911] req@00000000a4a2c44a x1846780895183872/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233936 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233911.729914:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895183872/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233936 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233911.729918:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233911.729919:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233911.729920:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895183872:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233912.497451:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233913.521400:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233914.545394:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233915.569396:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233916.593386:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233916.849338:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233916.849350:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233916.849355:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233916.849365:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6477 00000100:00080000:2.0:1761233916.849369:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233916.849375:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233916.849421:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895183936/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233916.849442:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233916.849446:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233916.849447:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233916.849451:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6477 00000100:00080000:2.0:1761233916.849453:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233916.849457:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233916.849472:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233916.849480:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895183936:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761233916.849488:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895184000/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233916.849496:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233916.849499:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233916.849500:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233916.849504:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6477 00000100:00080000:2.0:1761233916.849506:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233916.849509:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233916.849515:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233916.849523:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895184064/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233916.849529:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233916.849532:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233916.849533:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233916.849536:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6477 00000100:00080000:2.0:1761233916.849538:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233916.849540:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233916.849544:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761233916.849545:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233916.849551:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895184128/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233916.849557:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233916.849560:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233916.849561:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233916.849563:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6477 00000100:00080000:2.0:1761233916.849565:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761233916.849567:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233916/real 1761233916] req@000000004f29ca26 x1846780895183936/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233941 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233916.849569:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233916.849573:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233916.849580:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895184192/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233916.849583:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895183936/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233941 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233916.849588:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233916.849591:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233916.849592:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233916.849592:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233916.849595:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761233916.849596:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6477 00000100:00100000:1.0:1761233916.849598:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895183936:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761233916.849598:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233916.849601:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233916.849605:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233916.849611:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895184256/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233916.849617:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233916.849620:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895184000:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:2.0:1761233916.849620:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233916.849622:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233916.849625:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6477 00000100:00080000:2.0:1761233916.849627:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233916.849630:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233916.849633:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761233916.849641:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233916.849641:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895184320/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233916.849649:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233916/real 1761233916] req@000000007dcf84ae x1846780895184000/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233941 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233916.849659:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895184000/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233941 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233916.849673:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233916.849676:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233916.849679:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895184000:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233916.849685:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895184064:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233916.849704:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233916.849710:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233916/real 1761233916] req@0000000040d5f361 x1846780895184064/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233941 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233916.849717:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895184064/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233941 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233916.849722:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233916.849724:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233916.849725:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895184064:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233916.849728:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895184128:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233916.849738:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233916.849741:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233916/real 1761233916] req@000000009d315fa5 x1846780895184128/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233941 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233916.849747:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895184128/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233941 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233916.849752:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233916.849753:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233916.849754:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895184128:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233916.849758:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895184192:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233916.849767:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233916.849771:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233916/real 1761233916] req@0000000033765045 x1846780895184192/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233941 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233916.849777:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895184192/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233941 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233916.849782:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233916.849783:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233916.849785:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895184192:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233916.849788:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895184256:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233916.849797:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233916.849800:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233916/real 1761233916] req@0000000064a2cea0 x1846780895184256/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233941 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233916.849806:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895184256/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233941 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233916.849810:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233916.849811:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233916.849813:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895184256:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233916.849829:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895184320:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233916.849838:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233916.849842:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233916/real 1761233916] req@000000007091f547 x1846780895184320/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233941 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233916.849848:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895184320/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233941 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233916.849853:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233916.849854:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233916.849855:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895184320:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233917.617485:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233918.641364:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233919.665408:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233920.689396:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233921.713432:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233921.969335:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233921.969347:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233921.969352:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233921.969363:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6482 00000100:00080000:2.0:1761233921.969367:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233921.969374:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233921.969409:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895184384/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233921.969437:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233921.969442:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233921.969444:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233921.969448:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6482 00000100:00080000:2.0:1761233921.969451:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233921.969456:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233921.969467:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233921.969474:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895184384:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761233921.969482:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895184448/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233921.969494:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233921.969498:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233921.969499:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233921.969503:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6482 00000100:00080000:2.0:1761233921.969506:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233921.969510:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233921.969515:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233921.969527:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895184512/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233921.969535:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233921.969539:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233921.969541:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1761233921.969543:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233921.969548:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6482 00000100:00080000:2.0:1761233921.969551:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233921.969556:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233921.969561:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233921.969569:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233921/real 1761233921] req@00000000fd3935a6 x1846780895184384/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233946 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761233921.969578:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895184576/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233921.969590:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233921.969593:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895184384/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233946 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233921.969596:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233921.969599:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233921.969607:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6482 00000100:00080000:1.0:1761233921.969610:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233921.969610:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233921.969616:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761233921.969618:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761233921.969623:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895184384:192.168.201.157@tcp:250:kworker/u8:2.0 00000008:00100000:2.0:1761233921.969625:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233921.969640:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895184640/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233921.969651:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233921.969653:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895184448:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:2.0:1761233921.969658:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233921.969662:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233921.969667:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6482 00000100:00080000:2.0:1761233921.969670:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233921.969674:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233921.969679:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761233921.969683:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233921.969695:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895184704/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233921.969696:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233921/real 1761233921] req@00000000135e9a6e x1846780895184448/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233946 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233921.969719:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233921.969724:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895184448/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233946 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233921.969725:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233921.969727:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233921.969735:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6482 00000100:00080000:2.0:1761233921.969740:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233921.969746:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:1.0:1761233921.969748:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233921.969752:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000008:00100000:2.0:1761233921.969752:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233921.969758:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895184448:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233921.969768:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895184512:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761233921.969768:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895184768/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761233921.969793:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233921.969804:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233921/real 1761233921] req@000000006ff1fdbc x1846780895184512/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233946 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233921.969815:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895184512/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233946 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233921.969822:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233921.969825:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233921.969827:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895184512:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233921.969833:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895184576:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233921.969847:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233921.969854:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233921/real 1761233921] req@0000000088939481 x1846780895184576/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233946 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233921.969864:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895184576/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233946 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233921.969871:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233921.969873:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233921.969876:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895184576:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233921.969893:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895184640:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233921.969908:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233921.969915:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233921/real 1761233921] req@000000006d8c3eb0 x1846780895184640/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233946 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233921.969925:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895184640/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233946 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233921.969931:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233921.969934:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233921.969936:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895184640:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233921.969942:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895184704:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233921.969956:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233921.969962:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233921/real 1761233921] req@000000006c1041ba x1846780895184704/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233946 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233921.969971:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895184704/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233946 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233921.969978:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233921.969980:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233921.969983:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895184704:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233921.969988:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895184768:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233921.970001:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233921.970006:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233921/real 1761233921] req@00000000fd4aa3a2 x1846780895184768/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233946 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233921.970016:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895184768/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233946 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233921.970024:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233921.970026:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233921.970029:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895184768:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233922.737433:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233923.761391:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233924.785384:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233925.809414:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233926.833419:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233927.089346:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233927.089355:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233927.089358:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233927.089367:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6488 00000100:00080000:0.0:1761233927.089371:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233927.089377:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233927.089408:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895184832/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233927.089431:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233927.089434:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233927.089435:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233927.089437:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6488 00000100:00080000:0.0:1761233927.089439:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233927.089441:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233927.089451:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233927.089463:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895184896/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233927.089463:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895184832:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233927.089475:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233927.089477:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233927.089478:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233927.089480:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6488 00000100:00080000:0.0:1761233927.089481:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233927.089484:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233927.089487:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233927.089494:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895184960/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233927.089499:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233927.089501:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233927.089502:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233927.089504:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6488 00000100:00080000:0.0:1761233927.089505:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233927.089508:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233927.089511:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233927.089516:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895185024/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233927.089521:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233927.089523:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233927.089523:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233927.089527:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6488 00000100:00080000:0.0:1761233927.089529:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1761233927.089532:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233927.089535:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233927.089538:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233927.089546:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895185088/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233927.089551:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233927.089554:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1761233927.089555:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233927/real 1761233927] req@00000000d67bb589 x1846780895184832/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233952 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233927.089556:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233927.089564:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6488 00000100:00080000:0.0:1761233927.089567:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233927.089572:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233927.089575:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233927.089578:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895184832/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233952 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233927.089582:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895185152/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233927.089597:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761233927.089598:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233927.089602:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:0.0:1761233927.089603:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233927.089606:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761233927.089609:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895184832:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233927.089611:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6488 00000100:00080000:0.0:1761233927.089615:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233927.089617:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233927.089619:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233927.089625:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895185216/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233927.089633:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895184896:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233927.089654:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233927.089660:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233927/real 1761233927] req@000000009d1c7f73 x1846780895184896/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233952 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233927.089667:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895184896/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233952 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233927.089676:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233927.089678:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233927.089679:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895184896:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233927.089683:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895184960:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233927.089693:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233927.089697:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233927/real 1761233927] req@000000004130b6bd x1846780895184960/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233952 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233927.089703:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895184960/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233952 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233927.089707:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233927.089708:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233927.089710:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895184960:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233927.089713:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895185024:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233927.089723:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233927.089726:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233927/real 1761233927] req@00000000ef655201 x1846780895185024/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233952 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233927.089733:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895185024/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233952 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233927.089737:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233927.089738:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233927.089739:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895185024:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233927.089743:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895185088:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233927.089752:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233927.089756:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233927/real 1761233927] req@000000006d61dd06 x1846780895185088/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233952 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233927.089761:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895185088/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233952 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233927.089766:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233927.089767:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233927.089769:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895185088:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233927.089772:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895185152:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233927.089781:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233927.089784:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233927/real 1761233927] req@00000000efa29eb9 x1846780895185152/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233952 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233927.089791:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895185152/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233952 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233927.089796:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233927.089798:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233927.089801:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895185152:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233927.089815:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895185216:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233927.089825:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233927.089828:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233927/real 1761233927] req@000000001f9ec359 x1846780895185216/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233952 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233927.089835:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895185216/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233952 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233927.089840:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233927.089842:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233927.089844:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895185216:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233927.857520:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233928.881457:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233929.905390:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233930.929600:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233931.953380:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233932.209380:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233932.209398:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233932.209405:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233932.209420:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6493 00000100:00080000:0.0:1761233932.209425:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233932.209461:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233932.209547:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895185280/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233932.209582:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233932.209586:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233932.209589:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233932.209593:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6493 00000100:00080000:0.0:1761233932.209597:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233932.209602:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233932.209615:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233932.209625:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895185280:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761233932.209636:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895185344/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233932.209649:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233932.209653:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233932.209655:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233932.209660:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6493 00000100:00080000:0.0:1761233932.209663:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233932.209667:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233932.209676:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233932.209689:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895185408/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233932.209699:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233932.209703:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233932.209705:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1761233932.209708:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761233932.209713:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6493 00000100:00080000:0.0:1761233932.209716:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233932.209721:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233932.209726:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233932.209735:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233932/real 1761233932] req@000000005593c994 x1846780895185280/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233957 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761233932.209739:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895185472/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233932.209759:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233932.209762:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233932.209762:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233932.209767:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6493 00000100:00100000:1.0:1761233932.209767:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895185280/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233957 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761233932.209772:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233932.209779:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233932.209794:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1761233932.209798:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233932.209802:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761233932.209805:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895185536/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233932.209809:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895185280:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761233932.209821:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233932.209824:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233932.209826:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233932.209831:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6493 00000100:00080000:0.0:1761233932.209832:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233932.209834:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233932.209836:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233932.209844:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895185600/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233932.209847:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895185344:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:0.0:1761233932.209856:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233932.209859:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233932.209859:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233932.209863:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6493 00000100:00080000:0.0:1761233932.209865:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233932.209867:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233932.209870:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233932.209882:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895185664/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761233932.209887:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233932.209904:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233932/real 1761233932] req@00000000c6e06344 x1846780895185344/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233957 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233932.209918:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895185344/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233957 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233932.209930:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233932.209932:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233932.209934:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895185344:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233932.209939:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895185408:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233932.209959:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233932.209968:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233932/real 1761233932] req@000000004862b808 x1846780895185408/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233957 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233932.209981:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895185408/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233957 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233932.209989:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233932.209992:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233932.209996:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895185408:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233932.210002:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895185472:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233932.210025:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233932.210034:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233932/real 1761233932] req@000000006f2efc8b x1846780895185472/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233957 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233932.210047:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895185472/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233957 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233932.210056:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233932.210058:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233932.210062:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895185472:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233932.210067:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895185536:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233932.210084:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233932.210090:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233932/real 1761233932] req@00000000eca9ddc2 x1846780895185536/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233957 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233932.210098:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895185536/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233957 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233932.210103:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233932.210105:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233932.210107:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895185536:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233932.210121:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895185600:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233932.210133:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233932.210137:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233932/real 1761233932] req@0000000034d49f59 x1846780895185600/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233957 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233932.210144:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895185600/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233957 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233932.210149:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233932.210150:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233932.210187:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895185600:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233932.210192:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895185664:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233932.210244:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233932.210253:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233932/real 1761233932] req@000000005beaae31 x1846780895185664/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233957 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233932.210261:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895185664/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233957 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233932.210267:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233932.210269:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233932.210271:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895185664:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233932.977507:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233934.001332:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233935.025433:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233936.049411:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233937.073411:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233937.329369:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233937.329378:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233937.329382:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233937.329392:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6498 00000100:00080000:2.0:1761233937.329396:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233937.329401:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233937.329433:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895185728/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233937.329453:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233937.329455:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233937.329457:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233937.329459:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6498 00000100:00080000:2.0:1761233937.329460:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233937.329462:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233937.329469:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233937.329485:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895185792/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233937.329496:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233937.329500:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895185728:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761233937.329500:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233937.329502:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233937.329508:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6498 00000100:00080000:2.0:1761233937.329510:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233937.329512:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233937.329515:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233937.329522:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895185856/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233937.329528:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233937.329530:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233937.329531:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233937.329533:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6498 00000100:00080000:2.0:1761233937.329534:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233937.329536:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233937.329538:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233937.329546:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895185920/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233937.329550:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233937.329553:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233937.329554:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233937.329556:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6498 00000100:00080000:2.0:1761233937.329557:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233937.329560:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233937.329562:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233937.329567:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895185984/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233937.329573:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233937.329575:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233937.329576:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233937.329579:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6498 00000100:00080000:2.0:1761233937.329581:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1761233937.329584:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233937.329585:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233937.329590:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233937.329596:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895186048/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233937.329601:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233937.329603:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233937.329604:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233937.329607:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6498 00000100:00080000:2.0:1761233937.329608:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233937.329611:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761233937.329615:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233937/real 1761233937] req@00000000f4e415d3 x1846780895185728/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233962 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000008:00100000:2.0:1761233937.329615:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233937.329624:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895186112/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233937.329650:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895185728/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233962 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233937.329663:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233937.329667:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233937.329671:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895185728:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761233937.329696:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895185792:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233937.329722:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233937.329732:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233937/real 1761233937] req@0000000099b7e5d9 x1846780895185792/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233962 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233937.329743:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895185792/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233962 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233937.329756:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233937.329758:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233937.329761:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895185792:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233937.329767:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895185856:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233937.329787:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233937.329796:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233937/real 1761233937] req@00000000c10cebe6 x1846780895185856/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233962 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233937.329806:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895185856/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233962 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233937.329813:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233937.329815:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233937.329818:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895185856:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233937.329823:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895185920:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233937.329843:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233937.329851:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233937/real 1761233937] req@00000000a1e8c6bb x1846780895185920/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233962 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233937.329862:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895185920/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233962 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233937.329869:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233937.329872:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233937.329874:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895185920:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233937.329879:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895185984:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233937.329893:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233937.329898:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233937/real 1761233937] req@000000008ebccab5 x1846780895185984/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233962 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233937.329905:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895185984/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233962 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233937.329909:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233937.329911:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233937.329912:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895185984:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233937.329915:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895186048:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233937.329925:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233937.329928:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233937/real 1761233937] req@000000004ae5bc9b x1846780895186048/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233962 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233937.329934:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895186048/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233962 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233937.329939:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233937.329940:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233937.329941:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895186048:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233937.329944:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895186112:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233937.329954:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233937.329957:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233937/real 1761233937] req@00000000ac7d8d5b x1846780895186112/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233962 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233937.329964:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895186112/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233962 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233937.329968:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233937.329969:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233937.329971:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895186112:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233938.097445:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233939.121428:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233940.145445:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233941.169418:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233942.193406:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233942.449317:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233942.449326:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233942.449329:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233942.449340:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6503 00000100:00080000:2.0:1761233942.449343:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233942.449349:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233942.449383:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895186176/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233942.449402:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233942.449405:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233942.449407:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233942.449409:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6503 00000100:00080000:2.0:1761233942.449410:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233942.449413:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233942.449420:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233942.449429:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895186240/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233942.449434:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233942.449437:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233942.449437:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233942.449440:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6503 00000100:00080000:2.0:1761233942.449441:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233942.449443:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233942.449446:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233942.449449:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895186176:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761233942.449454:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895186304/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233942.449462:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233942.449464:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233942.449465:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233942.449468:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6503 00000100:00080000:2.0:1761233942.449470:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233942.449472:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233942.449474:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233942.449481:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895186368/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233942.449486:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233942.449488:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233942.449489:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233942.449492:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6503 00000100:00080000:2.0:1761233942.449493:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233942.449496:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233942.449498:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233942.449504:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895186432/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233942.449508:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233942.449510:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233942.449511:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233942.449513:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6503 00000100:00080000:2.0:1761233942.449514:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233942.449517:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233942.449519:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233942.449525:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895186496/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233942.449535:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233942.449538:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895186240:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:2.0:1761233942.449539:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233942.449542:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233942.449546:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6503 00000100:00080000:2.0:1761233942.449547:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233942.449550:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233942.449555:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233942.449563:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895186560/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233942.449578:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895186304:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233942.449620:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895186368:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233942.449640:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895186432:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233942.449658:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895186496:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233942.449679:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895186560:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233943.217423:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233944.241401:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233945.265458:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1761233945.521671:0:2209:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:0.0:1761233945.521725:0:2209:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1761233945.521745:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233945.521795:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233945.521807:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233945.521817:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233945.521828:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233945.521840:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761233945.521852:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:1.0:1761233945.521864:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233942/real 1761233945] req@000000008ebccab5 x1846780895186304/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233967 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233945.521897:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895186304/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233967 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233945.521921:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233945.521928:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233945.521934:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895186304:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233945.521993:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233942/real 1761233945] req@00000000a1e8c6bb x1846780895186368/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233967 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233945.522012:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895186368/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233967 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233945.522031:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233945.522034:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233945.522037:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895186368:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233945.522044:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233942/real 1761233945] req@00000000c10cebe6 x1846780895186432/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233967 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233945.522058:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895186432/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233967 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233945.522067:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233945.522070:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233945.522074:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895186432:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233945.522090:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233942/real 1761233945] req@0000000099b7e5d9 x1846780895186496/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233967 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233945.522105:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895186496/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233967 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233945.522114:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233945.522118:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233945.522121:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895186496:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233945.522128:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233942/real 1761233945] req@00000000f4e415d3 x1846780895186560/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233967 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233945.522157:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895186560/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233967 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233945.522167:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233945.522171:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233945.522175:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895186560:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233945.522182:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233942/real 1761233945] req@00000000ac7d8d5b x1846780895186176/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233967 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233945.522198:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895186176/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233967 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233945.522235:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233945.522238:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233945.522242:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895186176:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761233945.522249:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233942/real 1761233945] req@000000004ae5bc9b x1846780895186240/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233967 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233945.522263:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895186240/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233967 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233945.522271:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233945.522274:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233945.522277:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895186240:192.168.201.157@tcp:38:kworker/u8:2.0 00010000:00010000:1.0:1761233946.289395:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233947.313455:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233947.569397:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233947.569407:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233947.569411:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233947.569423:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6508 00000100:00080000:1.0:1761233947.569427:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233947.569434:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233947.569466:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090ac513e x1846780895186624/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233947.569491:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233947.569494:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233947.569496:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233947.569499:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6508 00000100:00080000:1.0:1761233947.569502:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233947.569506:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233947.569518:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233947.569533:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000053e52331 x1846780895186688/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233947.569546:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233947.569546:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895186624:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233947.569555:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233947.569558:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233947.569565:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6508 00000100:00080000:1.0:1761233947.569568:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233947.569572:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233947.569579:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233947.569595:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000880f16b4 x1846780895186752/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233947.569607:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233947.569611:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233947.569613:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233947.569618:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6508 00000100:00080000:1.0:1761233947.569621:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233947.569624:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233947.569630:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233947.569635:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233947.569649:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cb5be845 x1846780895186816/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233947.569662:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233947.569667:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233947/real 1761233947] req@0000000090ac513e x1846780895186624/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233972 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233947.569669:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233947.569672:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233947.569683:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6508 00000100:00080000:1.0:1761233947.569688:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233947.569704:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090ac513e x1846780895186624/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233972 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233947.569707:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233947.569717:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761233947.569726:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233947.569732:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233947.569736:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a95774fb x1846780895186880/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233947.569752:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895186624:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233947.569754:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233947.569764:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233947.569766:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233947.569774:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6508 00000100:00080000:1.0:1761233947.569777:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233947.569782:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233947.569788:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233947.569792:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895186688:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233947.569804:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000c736bb2 x1846780895186944/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233947.569817:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233947.569821:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233947.569823:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233947.569831:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6508 00000800:00000100:3.0:1761233947.569832:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:1.0:1761233947.569834:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233947.569850:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233947/real 1761233947] req@0000000053e52331 x1846780895186688/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233972 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233947.569852:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233947.569862:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233947.569876:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000053e52331 x1846780895186688/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233972 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233947.569884:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000038209f5 x1846780895187008/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233947.569898:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233947.569904:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233947.569933:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895186688:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233947.569943:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895186752:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233947.569973:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233947.569984:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233947/real 1761233947] req@00000000880f16b4 x1846780895186752/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233972 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233947.569998:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000880f16b4 x1846780895186752/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233972 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233947.570008:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233947.570011:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233947.570014:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895186752:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233947.570022:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895186816:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233947.570044:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233947.570053:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233947/real 1761233947] req@00000000cb5be845 x1846780895186816/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233972 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233947.570082:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cb5be845 x1846780895186816/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233972 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233947.570090:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233947.570093:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233947.570096:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895186816:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233947.570103:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895186880:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233947.570124:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233947.570132:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233947/real 1761233947] req@00000000a95774fb x1846780895186880/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233972 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233947.570144:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a95774fb x1846780895186880/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233972 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233947.570152:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233947.570155:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233947.570158:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895186880:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233947.570182:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895186944:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233947.570203:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233947.570212:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233947/real 1761233947] req@000000000c736bb2 x1846780895186944/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233972 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233947.570261:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000000c736bb2 x1846780895186944/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233972 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233947.570269:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233947.570271:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233947.570275:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895186944:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233947.570281:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895187008:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233947.570303:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233947.570313:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233947/real 1761233947] req@00000000038209f5 x1846780895187008/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233972 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233947.570326:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000038209f5 x1846780895187008/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233972 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233947.570334:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233947.570336:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233947.570340:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895187008:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233948.337444:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233949.361360:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233950.385359:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233951.409354:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233952.433439:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233952.689391:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233952.689404:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233952.689408:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233952.689420:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6513 00000100:00080000:2.0:1761233952.689424:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233952.689432:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233952.689470:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895187072/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233952.689499:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233952.689504:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233952.689505:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233952.689509:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6513 00000100:00080000:2.0:1761233952.689512:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233952.689516:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233952.689529:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233952.689542:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895187072:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761233952.689547:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895187136/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233952.689561:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233952.689565:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233952.689567:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233952.689571:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6513 00000100:00080000:2.0:1761233952.689573:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233952.689577:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233952.689582:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233952.689596:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895187200/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233952.689606:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233952.689610:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233952.689611:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233952.689615:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6513 00000100:00080000:2.0:1761233952.689619:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1761233952.689619:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233952.689626:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233952.689633:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233952.689644:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233952/real 1761233952] req@00000000fd4aa3a2 x1846780895187072/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233977 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761233952.689648:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895187264/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233952.689668:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233952.689672:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895187072/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233977 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233952.689675:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233952.689677:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233952.689685:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233952.689687:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6513 00000100:00080000:3.0:1761233952.689689:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761233952.689693:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233952.689696:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895187072:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761233952.689701:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233952.689708:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233952.689721:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895187328/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233952.689723:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895187136:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:2.0:1761233952.689738:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233952.689742:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233952.689744:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1761233952.689747:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233952.689753:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6513 00000100:00080000:2.0:1761233952.689756:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233952.689757:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233952/real 1761233952] req@000000006c1041ba x1846780895187136/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233977 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233952.689765:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233952.689774:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233952.689779:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895187136/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233977 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233952.689786:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233952.689788:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233952.689791:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895187392/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233952.689794:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895187136:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233952.689804:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895187200:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761233952.689815:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233952.689819:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233952.689821:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1761233952.689826:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761233952.689849:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6513 00000100:00080000:2.0:1761233952.689853:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233952.689856:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233952/real 1761233952] req@000000006d8c3eb0 x1846780895187200/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233977 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761233952.689861:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233952.689871:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233952.689874:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895187200/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233977 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233952.689881:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233952.689883:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761233952.689889:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895187456/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233952.689889:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895187200:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233952.689897:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895187264:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233952.689918:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233952.689927:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233952/real 1761233952] req@0000000088939481 x1846780895187264/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233977 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233952.689934:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895187264/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233977 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233952.689940:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233952.689942:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233952.689944:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895187264:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233952.689958:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895187328:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233952.689969:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233952.689973:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233952/real 1761233952] req@000000006ff1fdbc x1846780895187328/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233977 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233952.689980:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895187328/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233977 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233952.689985:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233952.689986:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233952.689987:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895187328:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233952.689991:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895187392:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233952.690001:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233952.690005:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233952/real 1761233952] req@00000000135e9a6e x1846780895187392/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233977 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233952.690011:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895187392/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233977 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233952.690016:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233952.690017:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233952.690019:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895187392:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233952.690022:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895187456:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233952.690033:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233952.690037:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233952/real 1761233952] req@00000000fd3935a6 x1846780895187456/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233977 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233952.690044:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895187456/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233977 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233952.690048:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233952.690049:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233952.690051:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895187456:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233953.457352:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233954.481384:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233955.505333:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233956.529444:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233957.553396:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761233957.809319:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233957.809332:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761233957.809336:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233957.809348:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6518 00000100:00080000:2.0:1761233957.809351:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233957.809359:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761233957.809395:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895187520/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233957.809420:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233957.809423:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233957.809425:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233957.809428:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6518 00000100:00080000:2.0:1761233957.809430:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233957.809433:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233957.809441:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233957.809454:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895187584/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233957.809460:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895187520:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761233957.809479:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233957.809483:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233957.809484:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233957.809488:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6518 00000100:00080000:2.0:1761233957.809490:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233957.809494:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233957.809501:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233957.809513:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895187648/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233957.809520:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233957.809522:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233957.809523:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233957.809526:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6518 00000100:00080000:2.0:1761233957.809528:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233957.809530:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233957.809534:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761233957.809544:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761233957.809548:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895187712/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233957.809554:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233957.809557:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761233957.809559:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233957.809563:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6518 00000100:00080000:2.0:1761233957.809565:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761233957.809569:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233957/real 1761233957] req@000000007091f547 x1846780895187520/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233982 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233957.809570:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233957.809576:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761233957.809586:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895187776/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233957.809593:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895187520/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233982 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761233957.809593:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233957.809599:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761233957.809601:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233957.809609:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6518 00000100:00080000:1.0:1761233957.809612:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761233957.809612:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233957.809617:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:1.0:1761233957.809618:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000008:00100000:2.0:1761233957.809623:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233957.809627:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895187520:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761233957.809633:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895187840/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761233957.809640:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761233957.809643:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761233957.809644:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761233957.809646:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6518 00000100:00080000:2.0:1761233957.809647:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761233957.809650:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761233957.809653:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233957.809655:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895187584:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761233957.809663:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895187904/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761233957.809680:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233957.809690:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233957/real 1761233957] req@0000000064a2cea0 x1846780895187584/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233982 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233957.809699:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895187584/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233982 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233957.809704:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233957.809705:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233957.809707:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895187584:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233957.809712:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895187648:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233957.809724:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233957.809729:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233957/real 1761233957] req@0000000033765045 x1846780895187648/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233982 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233957.809736:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895187648/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233982 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233957.809741:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233957.809743:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233957.809744:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895187648:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233957.809748:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895187712:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233957.809784:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233957.809789:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233957/real 1761233957] req@000000009d315fa5 x1846780895187712/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233982 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233957.809796:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895187712/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233982 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233957.809802:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233957.809803:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233957.809805:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895187712:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233957.809809:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895187776:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233957.809820:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233957.809824:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233957/real 1761233957] req@0000000040d5f361 x1846780895187776/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233982 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233957.809831:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895187776/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233982 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233957.809837:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233957.809838:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233957.809840:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895187776:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233957.809843:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895187840:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233957.809854:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233957.809859:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233957/real 1761233957] req@000000007dcf84ae x1846780895187840/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233982 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233957.809865:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895187840/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233982 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233957.809871:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233957.809872:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233957.809874:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895187840:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233957.809890:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895187904:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233957.809901:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233957.809906:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233957/real 1761233957] req@000000004f29ca26 x1846780895187904/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233982 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233957.809913:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895187904/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233982 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233957.809918:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233957.809920:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233957.809921:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895187904:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761233958.577398:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233959.601376:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233960.625525:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233961.649422:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233962.673411:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761233962.929370:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233962.929380:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761233962.929383:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233962.929391:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6523 00000100:00080000:0.0:1761233962.929394:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233962.929400:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761233962.929430:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000addc3db8 x1846780895187968/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233962.929453:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233962.929456:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233962.929458:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233962.929460:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6523 00000100:00080000:0.0:1761233962.929462:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233962.929464:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233962.929471:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233962.929478:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000078e7052d x1846780895188032/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233962.929483:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233962.929486:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233962.929487:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233962.929489:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6523 00000100:00080000:0.0:1761233962.929490:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233962.929492:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233962.929496:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233962.929503:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005beaae31 x1846780895188096/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233962.929510:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233962.929512:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233962.929514:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761233962.929515:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895187968:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761233962.929520:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6523 00000100:00080000:0.0:1761233962.929523:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233962.929525:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233962.929528:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233962.929536:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000034d49f59 x1846780895188160/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233962.929542:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233962.929544:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761233962.929545:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233962.929547:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6523 00000100:00080000:0.0:1761233962.929548:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233962.929551:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233962.929553:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761233962.929559:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000eca9ddc2 x1846780895188224/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233962.929564:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233962.929566:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761233962.929567:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233962.929569:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6523 00000100:00080000:0.0:1761233962.929570:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761233962.929574:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233962.929578:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761233962.929585:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761233962.929590:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006f2efc8b x1846780895188288/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761233962.929608:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761233962.929612:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761233962.929613:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761233962.929618:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6523 00000100:00080000:0.0:1761233962.929621:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761233962.929622:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233962/real 1761233962] req@00000000addc3db8 x1846780895187968/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233987 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761233962.929628:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761233962.929637:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233962.929641:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000addc3db8 x1846780895187968/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233987 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:0.0:1761233962.929648:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004862b808 x1846780895188352/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233962.929651:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233962.929656:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233962.929663:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000addc3db8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895187968:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761233962.929689:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895188032:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233962.929723:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233962.929730:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233962/real 1761233962] req@0000000078e7052d x1846780895188032/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233987 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233962.929738:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000078e7052d x1846780895188032/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233987 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233962.929743:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233962.929744:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233962.929746:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000078e7052d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895188032:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233962.929765:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895188096:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233962.929777:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233962.929781:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233962/real 1761233962] req@000000005beaae31 x1846780895188096/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233987 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233962.929788:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005beaae31 x1846780895188096/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233987 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233962.929793:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233962.929794:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233962.929796:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005beaae31 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895188096:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233962.929799:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895188160:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233962.929809:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233962.929813:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233962/real 1761233962] req@0000000034d49f59 x1846780895188160/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233987 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233962.929820:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000034d49f59 x1846780895188160/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233987 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233962.929824:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233962.929826:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233962.929827:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000034d49f59 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895188160:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233962.929830:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895188224:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761233962.929840:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233962.929844:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233962/real 1761233962] req@00000000eca9ddc2 x1846780895188224/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233987 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233962.929850:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000eca9ddc2 x1846780895188224/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233987 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233962.929855:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233962.929856:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233962.929858:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000eca9ddc2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895188224:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761233962.929861:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895188288:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233962.929871:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233962.929875:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233962/real 1761233962] req@000000006f2efc8b x1846780895188288/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233987 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233962.929881:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006f2efc8b x1846780895188288/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233987 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233962.929890:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233962.929892:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233962.929894:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006f2efc8b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895188288:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761233962.929897:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895188352:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761233962.929907:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233962.929911:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233962/real 1761233962] req@000000004862b808 x1846780895188352/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233987 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761233962.929917:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004862b808 x1846780895188352/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233987 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761233962.929921:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233962.929922:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233962.929924:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004862b808 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895188352:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761233963.697452:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233964.721588:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233965.745364:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233966.769460:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233967.793371:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761233968.049338:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233968.049350:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761233968.049354:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233968.049366:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6528 00000100:00080000:3.0:1761233968.049370:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233968.049377:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761233968.049410:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007aee79ba x1846780895188416/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233968.049432:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233968.049435:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233968.049436:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233968.049439:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6528 00000100:00080000:3.0:1761233968.049440:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233968.049443:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233968.049451:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233968.049460:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009706a782 x1846780895188480/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233968.049464:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895188416:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:3.0:1761233968.049470:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233968.049473:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233968.049473:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233968.049476:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6528 00000100:00080000:3.0:1761233968.049478:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233968.049480:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233968.049483:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233968.049491:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000bbf6af7b x1846780895188544/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233968.049496:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233968.049498:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233968.049499:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233968.049501:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6528 00000100:00080000:3.0:1761233968.049502:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233968.049504:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233968.049508:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233968.049515:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e55ffa x1846780895188608/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233968.049519:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233968.049521:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233968.049523:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233968.049525:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6528 00000100:00080000:3.0:1761233968.049527:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1761233968.049530:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761233968.049531:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233968.049535:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233968.049542:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a0ba65f x1846780895188672/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233968.049548:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233968.049550:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233968.049552:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761233968.049554:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233968/real 1761233968] req@000000007aee79ba x1846780895188416/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233993 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233968.049557:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6528 00000100:00080000:3.0:1761233968.049561:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233968.049566:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233968.049569:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233968.049576:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895188416/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233993 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233968.049577:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c9993cd4 x1846780895188736/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233968.049605:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233968.049605:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233968.049611:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233968.049612:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:3.0:1761233968.049613:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233968.049618:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6528 00000100:00100000:1.0:1761233968.049621:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895188416:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:3.0:1761233968.049622:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233968.049627:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233968.049633:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233968.049641:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002625ec0b x1846780895188800/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233968.049656:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895188480:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233968.049680:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233968.049689:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233968/real 1761233968] req@000000009706a782 x1846780895188480/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233993 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233968.049699:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895188480/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233993 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233968.049706:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233968.049708:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233968.049711:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895188480:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233968.049716:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895188544:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233968.049733:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233968.049740:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233968/real 1761233968] req@00000000bbf6af7b x1846780895188544/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233993 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233968.049749:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895188544/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233993 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233968.049755:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233968.049757:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233968.049760:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895188544:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233968.049765:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895188608:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233968.049782:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233968.049790:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233968/real 1761233968] req@0000000057e55ffa x1846780895188608/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233993 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233968.049801:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895188608/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233993 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233968.049808:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233968.049811:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233968.049814:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895188608:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233968.049819:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895188672:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233968.049838:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233968.049845:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233968/real 1761233968] req@000000002a0ba65f x1846780895188672/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233993 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233968.049856:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895188672/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233993 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233968.049864:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233968.049866:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233968.049869:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895188672:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233968.049875:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895188736:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233968.049891:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233968.049896:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233968/real 1761233968] req@00000000c9993cd4 x1846780895188736/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233993 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233968.049903:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895188736/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233993 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233968.049911:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233968.049913:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233968.049915:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895188736:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233968.049918:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895188800:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233968.049927:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233968.049931:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233968/real 1761233968] req@000000002625ec0b x1846780895188800/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233993 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233968.049937:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895188800/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233993 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233968.049941:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233968.049943:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233968.049944:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895188800:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233968.817410:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233969.841383:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233970.865402:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233971.889478:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233972.913452:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761233973.169359:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233973.169367:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761233973.169371:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233973.169381:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6534 00000100:00080000:3.0:1761233973.169384:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233973.169390:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761233973.169424:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b40830d x1846780895188864/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233973.169446:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233973.169449:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233973.169450:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233973.169453:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6534 00000100:00080000:3.0:1761233973.169455:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233973.169458:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233973.169466:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233973.169475:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000829a5595 x1846780895188928/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233973.169503:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895188864:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:3.0:1761233973.169506:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233973.169514:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233973.169516:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233973.169522:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6534 00000100:00080000:3.0:1761233973.169524:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233973.169528:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233973.169534:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233973.169548:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000089b8c4f1 x1846780895188992/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233973.169559:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233973.169562:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233973.169564:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233973.169569:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6534 00000100:00080000:3.0:1761233973.169571:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233973.169574:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233973.169579:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761233973.169584:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233973.169592:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000070fa3bd6 x1846780895189056/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233973.169600:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233973.169604:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233973.169605:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233973.169608:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6534 00000100:00080000:3.0:1761233973.169611:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761233973.169613:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233973/real 1761233973] req@000000003b40830d x1846780895188864/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233998 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233973.169617:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233973.169623:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233973.169632:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090702bc4 x1846780895189120/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233973.169639:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233973.169643:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895188864/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761233998 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233973.169644:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233973.169646:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233973.169653:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6534 00000100:00080000:3.0:1761233973.169656:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233973.169660:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233973.169665:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1761233973.169667:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1761233973.169672:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6fdd7f9 x1846780895189184/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233973.169673:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:3.0:1761233973.169682:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233973.169683:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895188864:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:3.0:1761233973.169686:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233973.169688:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233973.169694:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6534 00000100:00080000:3.0:1761233973.169695:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233973.169698:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233973.169700:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233973.169707:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008e58c4b8 x1846780895189248/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233973.169719:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895188928:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233973.169744:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233973.169752:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233973/real 1761233973] req@00000000829a5595 x1846780895188928/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233998 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233973.169760:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895188928/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233998 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233973.169766:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233973.169768:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233973.169770:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895188928:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233973.169774:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895188992:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233973.169785:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233973.169790:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233973/real 1761233973] req@0000000089b8c4f1 x1846780895188992/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233998 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233973.169796:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895188992/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233998 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233973.169801:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233973.169803:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233973.169804:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895188992:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233973.169808:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895189056:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233973.169818:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233973.169823:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233973/real 1761233973] req@0000000070fa3bd6 x1846780895189056/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233998 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233973.169829:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895189056/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233998 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233973.169834:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233973.169836:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233973.169838:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895189056:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233973.169841:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895189120:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233973.169851:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233973.169855:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233973/real 1761233973] req@0000000090702bc4 x1846780895189120/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233998 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233973.169862:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895189120/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761233998 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233973.169867:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233973.169869:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233973.169872:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895189120:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233973.169878:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895189184:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233973.169897:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233973.169905:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233973/real 1761233973] req@00000000b6fdd7f9 x1846780895189184/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233998 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233973.169917:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895189184/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233998 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233973.169931:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233973.169933:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233973.169935:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895189184:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233973.169969:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895189248:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233973.169983:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233973.169987:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233973/real 1761233973] req@000000008e58c4b8 x1846780895189248/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233998 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233973.169994:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895189248/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761233998 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233973.169998:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233973.169999:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233973.170001:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895189248:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233973.937588:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233974.961386:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233975.985492:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233977.009539:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233978.033459:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761233978.289285:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233978.289297:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761233978.289302:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233978.289314:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6539 00000100:00080000:3.0:1761233978.289318:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233978.289325:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761233978.289367:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000080131ea7 x1846780895189312/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233978.289405:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233978.289408:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233978.289410:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233978.289413:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6539 00000100:00080000:3.0:1761233978.289415:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233978.289419:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233978.289430:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233978.289445:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007adf99ab x1846780895189376/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233978.289452:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895189312:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:3.0:1761233978.289459:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233978.289465:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233978.289467:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233978.289472:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6539 00000100:00080000:3.0:1761233978.289475:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233978.289482:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233978.289488:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233978.289503:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e99e5a12 x1846780895189440/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233978.289512:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233978.289516:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233978.289518:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233978.289522:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6539 00000100:00080000:3.0:1761233978.289524:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1761233978.289531:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761233978.289532:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233978.289541:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233978.289556:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233978/real 1761233978] req@0000000080131ea7 x1846780895189312/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234003 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233978.289562:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008e58c4b8 x1846780895189504/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233978.289596:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233978.289597:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895189312/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234003 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233978.289615:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233978.289617:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233978.289627:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6539 00000100:00080000:1.0:1761233978.289629:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233978.289630:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233978.289636:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:3.0:1761233978.289640:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761233978.289644:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895189312:192.168.201.157@tcp:250:kworker/u8:3.0 00000008:00100000:3.0:1761233978.289646:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233978.289664:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6fdd7f9 x1846780895189568/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233978.289676:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233978.289680:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895189376:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:3.0:1761233978.289681:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233978.289686:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233978.289694:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6539 00000100:00080000:3.0:1761233978.289697:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1761233978.289704:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761233978.289708:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761233978.289716:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233978/real 1761233978] req@000000007adf99ab x1846780895189376/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234003 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:3.0:1761233978.289717:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233978.289732:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895189376/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234003 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233978.289740:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233978.289743:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233978.289744:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090702bc4 x1846780895189632/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761233978.289750:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895189376:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233978.289769:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895189440:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1761233978.289785:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233978.289791:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1761233978.289794:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761233978.289795:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761233978.289801:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233978/real 1761233978] req@00000000e99e5a12 x1846780895189440/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234003 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233978.289807:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6539 00000100:00080000:3.0:1761233978.289815:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761233978.289823:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895189440/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234003 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233978.289827:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:1.0:1761233978.289839:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233978.289843:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000008:00100000:3.0:1761233978.289843:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233978.289848:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895189440:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233978.289855:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895189504:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233978.289870:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000070fa3bd6 x1846780895189696/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761233978.289876:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233978.289883:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233978/real 1761233978] req@000000008e58c4b8 x1846780895189504/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234003 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233978.289920:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895189504/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234003 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233978.289927:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233978.289928:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233978.289930:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895189504:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233978.289938:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895189568:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233978.289951:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233978.289956:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233978/real 1761233978] req@00000000b6fdd7f9 x1846780895189568/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234003 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233978.289962:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895189568/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234003 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233978.289967:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233978.289969:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233978.289970:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895189568:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233978.289987:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895189632:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233978.289998:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233978.290002:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233978/real 1761233978] req@0000000090702bc4 x1846780895189632/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234003 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233978.290009:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895189632/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234003 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233978.290018:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233978.290020:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233978.290022:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895189632:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233978.290026:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895189696:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233978.290036:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233978.290041:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233978/real 1761233978] req@0000000070fa3bd6 x1846780895189696/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234003 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233978.290047:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895189696/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234003 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233978.290052:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233978.290053:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233978.290055:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895189696:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233979.057422:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233980.081447:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233981.105396:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233982.129441:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233983.153338:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761233983.409273:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233983.409281:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761233983.409285:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233983.409307:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6544 00000100:00080000:3.0:1761233983.409311:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233983.409317:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761233983.409348:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000089b8c4f1 x1846780895189760/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233983.409370:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233983.409373:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233983.409374:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233983.409376:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6544 00000100:00080000:3.0:1761233983.409378:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233983.409380:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233983.409387:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233983.409396:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000829a5595 x1846780895189824/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233983.409402:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233983.409404:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233983.409406:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761233983.409408:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895189760:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:3.0:1761233983.409411:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6544 00000100:00080000:3.0:1761233983.409414:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233983.409418:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233983.409422:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233983.409429:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b40830d x1846780895189888/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233983.409434:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233983.409437:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233983.409437:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233983.409440:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6544 00000100:00080000:3.0:1761233983.409441:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233983.409444:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233983.409447:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233983.409453:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002625ec0b x1846780895189952/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233983.409458:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233983.409461:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761233983.409462:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233983.409464:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6544 00000100:00080000:3.0:1761233983.409465:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233983.409468:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233983.409471:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233983.409478:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c9993cd4 x1846780895190016/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761233983.409485:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761233983.409487:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761233983.409490:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761233983.409491:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233983.409494:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6544 00000100:00080000:3.0:1761233983.409496:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233983.409499:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233983.409501:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761233983.409507:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a0ba65f x1846780895190080/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761233983.409512:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761233983.409516:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233983/real 1761233983] req@0000000089b8c4f1 x1846780895189760/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234008 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233983.409516:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761233983.409518:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761233983.409524:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6544 00000100:00080000:3.0:1761233983.409527:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233983.409530:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761233983.409533:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233983.409540:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895189760/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234008 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233983.409548:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e55ffa x1846780895190144/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233983.409549:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233983.409554:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233983.409560:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895189760:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761233983.409582:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895189824:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233983.409599:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233983.409604:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233983/real 1761233983] req@00000000829a5595 x1846780895189824/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234008 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233983.409611:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895189824/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234008 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233983.409616:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233983.409618:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233983.409620:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895189824:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233983.409623:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895189888:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233983.409634:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233983.409638:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233983/real 1761233983] req@000000003b40830d x1846780895189888/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234008 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233983.409644:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895189888/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234008 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233983.409648:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233983.409649:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233983.409651:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895189888:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233983.409654:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895189952:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233983.409663:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233983.409667:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233983/real 1761233983] req@000000002625ec0b x1846780895189952/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234008 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233983.409673:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895189952/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234008 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233983.409678:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233983.409679:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233983.409680:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895189952:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233983.409683:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895190016:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761233983.409692:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233983.409696:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233983/real 1761233983] req@00000000c9993cd4 x1846780895190016/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234008 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233983.409701:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895190016/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234008 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233983.409706:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233983.409707:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233983.409709:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895190016:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761233983.409712:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895190080:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233983.409721:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233983.409724:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233983/real 1761233983] req@000000002a0ba65f x1846780895190080/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234008 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233983.409730:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895190080/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234008 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233983.409738:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233983.409739:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233983.409741:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895190080:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761233983.409744:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895190144:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761233983.409753:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233983.409756:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233983/real 1761233983] req@0000000057e55ffa x1846780895190144/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234008 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761233983.409762:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895190144/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234008 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233983.409766:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233983.409767:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761233983.409769:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895190144:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233984.177444:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233985.201439:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233986.225396:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233987.249512:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233988.273391:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233988.529365:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233988.529375:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233988.529378:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233988.529387:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6549 00000100:00080000:1.0:1761233988.529391:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233988.529397:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233988.529430:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000568f2a96 x1846780895190208/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233988.529453:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233988.529457:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233988.529458:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233988.529461:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6549 00000100:00080000:1.0:1761233988.529463:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233988.529466:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233988.529475:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233988.529483:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c652d0ad x1846780895190272/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233988.529490:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233988.529493:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1761233988.529494:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895190208:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233988.529495:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233988.529501:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6549 00000100:00080000:1.0:1761233988.529503:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233988.529505:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233988.529508:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233988.529517:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ba3340f x1846780895190336/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233988.529522:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233988.529524:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233988.529525:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233988.529528:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6549 00000100:00080000:1.0:1761233988.529529:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233988.529531:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233988.529533:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233988.529539:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d74b2e3c x1846780895190400/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233988.529544:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233988.529546:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233988.529547:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233988.529550:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6549 00000100:00080000:1.0:1761233988.529551:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233988.529553:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233988.529556:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761233988.529559:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761233988.529566:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001aedf9b1 x1846780895190464/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233988.529573:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233988.529575:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233988.529577:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233988.529579:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6549 00000100:00080000:1.0:1761233988.529580:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761233988.529582:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233988/real 1761233988] req@00000000568f2a96 x1846780895190208/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234013 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233988.529586:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233988.529591:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233988.529599:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a611ee34 x1846780895190528/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233988.529605:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233988.529605:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000568f2a96 x1846780895190208/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234013 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761233988.529611:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233988.529613:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233988.529619:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6549 00000100:00080000:3.0:1761233988.529620:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761233988.529622:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761233988.529624:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:1.0:1761233988.529627:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233988.529630:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895190208:192.168.201.157@tcp:250:kworker/u8:3.0 00000008:00100000:1.0:1761233988.529631:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233988.529640:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000328368c8 x1846780895190592/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233988.529656:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895190272:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233988.529673:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233988.529678:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233988/real 1761233988] req@00000000c652d0ad x1846780895190272/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234013 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233988.529686:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c652d0ad x1846780895190272/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234013 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233988.529691:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233988.529692:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233988.529694:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895190272:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233988.529697:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895190336:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233988.529708:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233988.529711:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233988/real 1761233988] req@000000007ba3340f x1846780895190336/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234013 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233988.529717:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007ba3340f x1846780895190336/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234013 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233988.529721:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233988.529723:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233988.529724:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895190336:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233988.529727:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895190400:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233988.529736:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233988.529740:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233988/real 1761233988] req@00000000d74b2e3c x1846780895190400/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234013 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233988.529745:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d74b2e3c x1846780895190400/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234013 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233988.529758:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233988.529760:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233988.529762:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895190400:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233988.529768:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895190464:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233988.529781:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233988.529785:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233988/real 1761233988] req@000000001aedf9b1 x1846780895190464/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234013 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233988.529792:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001aedf9b1 x1846780895190464/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234013 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233988.529796:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233988.529797:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233988.529799:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895190464:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233988.529802:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895190528:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233988.529812:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233988.529816:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233988/real 1761233988] req@00000000a611ee34 x1846780895190528/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234013 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233988.529822:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a611ee34 x1846780895190528/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234013 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233988.529826:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233988.529827:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233988.529829:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895190528:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233988.529832:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895190592:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233988.529842:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233988.529846:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233988/real 1761233988] req@00000000328368c8 x1846780895190592/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234013 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233988.529852:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000328368c8 x1846780895190592/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234013 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233988.529856:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233988.529857:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233988.529859:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895190592:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233989.297423:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233990.321398:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233991.345453:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233992.369408:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233993.393443:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233993.649370:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233993.649383:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233993.649388:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233993.649399:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6554 00000100:00080000:1.0:1761233993.649404:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233993.649414:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233993.649460:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090800dde x1846780895190656/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233993.649481:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233993.649484:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233993.649485:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233993.649488:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6554 00000100:00080000:1.0:1761233993.649489:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233993.649491:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233993.649500:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233993.649512:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e467e107 x1846780895190720/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233993.649518:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233993.649524:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233993.649525:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233993.649528:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6554 00000100:00080000:1.0:1761233993.649530:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233993.649534:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233993.649539:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233993.649547:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cbad49de x1846780895190784/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233993.649552:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761233993.649554:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090800dde pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895190656:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761233993.649556:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233993.649558:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233993.649564:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6554 00000100:00080000:1.0:1761233993.649566:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233993.649568:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233993.649570:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233993.649580:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000328368c8 x1846780895190848/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233993.649585:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233993.649587:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233993.649588:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233993.649591:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6554 00000100:00080000:1.0:1761233993.649593:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233993.649595:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233993.649598:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233993.649605:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a611ee34 x1846780895190912/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233993.649610:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233993.649613:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233993.649613:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233993.649615:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6554 00000100:00080000:1.0:1761233993.649616:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233993.649619:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233993.649621:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233993.649627:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001aedf9b1 x1846780895190976/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233993.649632:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233993.649635:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233993.649636:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1761233993.649640:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:1.0:1761233993.649642:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6554 00000100:00080000:1.0:1761233993.649644:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233993.649647:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233993.649650:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233993.649658:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d74b2e3c x1846780895191040/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761233993.649670:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233993/real 1761233993] req@0000000090800dde x1846780895190656/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234018 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233993.649687:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090800dde x1846780895190656/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234018 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233993.649695:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233993.649697:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233993.649701:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090800dde pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895190656:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233993.649706:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e467e107 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895190720:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233993.649722:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233993.649727:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233993/real 1761233993] req@00000000e467e107 x1846780895190720/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234018 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233993.649734:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e467e107 x1846780895190720/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234018 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233993.649739:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233993.649741:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233993.649743:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e467e107 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895190720:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233993.649777:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbad49de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895190784:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233993.649790:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233993.649795:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233993/real 1761233993] req@00000000cbad49de x1846780895190784/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234018 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233993.649801:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cbad49de x1846780895190784/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234018 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233993.649806:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233993.649808:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233993.649810:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbad49de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895190784:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233993.649813:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895190848:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233993.649822:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233993.649826:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233993/real 1761233993] req@00000000328368c8 x1846780895190848/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234018 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233993.649832:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000328368c8 x1846780895190848/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234018 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233993.649841:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233993.649842:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233993.649844:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895190848:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233993.649847:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895190912:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233993.649857:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233993.649860:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233993/real 1761233993] req@00000000a611ee34 x1846780895190912/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234018 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233993.649867:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a611ee34 x1846780895190912/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234018 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233993.649871:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233993.649872:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233993.649874:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895190912:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233993.649877:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895190976:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233993.649886:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233993.649890:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233993/real 1761233993] req@000000001aedf9b1 x1846780895190976/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234018 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233993.649895:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001aedf9b1 x1846780895190976/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234018 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233993.649899:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233993.649900:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233993.649902:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895190976:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233993.649906:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895191040:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233993.649916:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233993.649920:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233993/real 1761233993] req@00000000d74b2e3c x1846780895191040/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234018 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233993.649926:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d74b2e3c x1846780895191040/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234018 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233993.649931:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233993.649932:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233993.649933:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895191040:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761233994.417397:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761233995.441442:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233996.465422:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233997.489470:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761233998.513418:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761233998.769350:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233998.769359:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761233998.769364:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233998.769376:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6559 00000100:00080000:1.0:1761233998.769380:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233998.769387:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761233998.769424:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ba3340f x1846780895191104/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233998.769450:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233998.769453:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233998.769454:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233998.769457:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6559 00000100:00080000:1.0:1761233998.769458:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233998.769460:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233998.769468:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233998.769475:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c652d0ad x1846780895191168/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233998.769480:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233998.769483:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233998.769484:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233998.769487:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6559 00000100:00080000:1.0:1761233998.769489:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233998.769491:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233998.769494:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233998.769499:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000568f2a96 x1846780895191232/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233998.769505:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233998.769507:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233998.769509:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233998.769511:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6559 00000100:00080000:1.0:1761233998.769512:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233998.769517:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761233998.769517:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895191104:192.168.201.157@tcp:250:kworker/u8:3.0 00000008:00100000:1.0:1761233998.769521:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233998.769528:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000038209f5 x1846780895191296/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233998.769533:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233998.769535:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761233998.769536:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233998.769539:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6559 00000100:00080000:1.0:1761233998.769541:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233998.769543:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233998.769546:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233998.769552:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000c736bb2 x1846780895191360/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233998.769557:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233998.769559:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761233998.769560:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233998.769562:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6559 00000100:00080000:1.0:1761233998.769563:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233998.769566:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233998.769569:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233998.769574:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a95774fb x1846780895191424/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761233998.769579:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761233998.769581:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761233998.769583:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761233998.769585:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6559 00000100:00080000:1.0:1761233998.769586:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761233998.769588:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761233998.769590:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761233998.769597:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cb5be845 x1846780895191488/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761233998.769601:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233998.769628:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233998/real 1761233998] req@000000007ba3340f x1846780895191104/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234023 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233998.769646:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007ba3340f x1846780895191104/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234023 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233998.769657:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233998.769660:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233998.769664:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895191104:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761233998.769671:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895191168:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233998.769692:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233998.769698:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233998/real 1761233998] req@00000000c652d0ad x1846780895191168/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234023 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233998.769710:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c652d0ad x1846780895191168/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234023 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233998.769717:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233998.769719:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233998.769722:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895191168:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233998.769748:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895191232:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233998.769766:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233998.769772:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233998/real 1761233998] req@00000000568f2a96 x1846780895191232/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234023 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233998.769782:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000568f2a96 x1846780895191232/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234023 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233998.769790:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233998.769792:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233998.769795:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895191232:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233998.769800:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895191296:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233998.769814:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233998.769820:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233998/real 1761233998] req@00000000038209f5 x1846780895191296/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234023 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233998.769830:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000038209f5 x1846780895191296/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234023 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233998.769838:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233998.769840:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233998.769843:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895191296:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233998.769847:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895191360:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761233998.769863:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233998.769869:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233998/real 1761233998] req@000000000c736bb2 x1846780895191360/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234023 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233998.769879:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000000c736bb2 x1846780895191360/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234023 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233998.769887:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233998.769889:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233998.769891:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895191360:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761233998.769896:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895191424:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233998.769911:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233998.769917:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233998/real 1761233998] req@00000000a95774fb x1846780895191424/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234023 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233998.769928:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a95774fb x1846780895191424/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234023 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233998.769935:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233998.769937:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233998.769940:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895191424:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761233998.769945:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895191488:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761233998.769960:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761233998.769966:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761233998/real 1761233998] req@00000000cb5be845 x1846780895191488/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234023 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761233998.769975:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cb5be845 x1846780895191488/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234023 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761233998.769982:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761233998.769984:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761233998.769987:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895191488:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761233999.537464:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234000.561418:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234001.585369:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234002.609498:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234003.633460:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761234003.889375:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234003.889394:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761234003.889401:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234003.889418:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6564 00000100:00080000:3.0:1761234003.889422:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234003.889430:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761234003.889480:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000bbf6af7b x1846780895191552/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234003.889512:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234003.889516:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234003.889518:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234003.889535:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6564 00000100:00080000:3.0:1761234003.889539:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234003.889545:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234003.889556:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234003.889562:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895191552:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761234003.889577:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009706a782 x1846780895191616/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234003.889593:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234003.889598:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234003.889600:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234003.889604:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6564 00000100:00080000:3.0:1761234003.889607:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234003.889611:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234003.889618:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234003.889630:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007aee79ba x1846780895191680/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234003.889645:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1761234003.889646:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761234003.889654:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234003.889656:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234003.889661:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6564 00000100:00080000:3.0:1761234003.889664:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234003.889667:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234003.889671:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234003.889675:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234003/real 1761234003] req@00000000bbf6af7b x1846780895191552/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234028 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234003.889685:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e55ffa x1846780895191744/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761234003.889692:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895191552/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234028 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234003.889694:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234003.889706:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234003.889710:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234003.889719:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6564 00000100:00080000:1.0:1761234003.889721:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234003.889722:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234003.889732:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:3.0:1761234003.889734:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761234003.889741:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895191552:192.168.201.157@tcp:250:kworker/u8:3.0 00000008:00100000:3.0:1761234003.889742:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234003.889760:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a0ba65f x1846780895191808/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234003.889766:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234003.889769:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234003.889770:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234003.889775:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6564 00000100:00100000:1.0:1761234003.889776:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895191616:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:3.0:1761234003.889780:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234003.889793:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234003.889796:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761234003.889798:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234003.889808:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234003/real 1761234003] req@000000009706a782 x1846780895191616/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234028 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234003.889814:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c9993cd4 x1846780895191872/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761234003.889834:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895191616/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234028 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234003.889836:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234003.889850:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234003.889853:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234003.889858:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234003.889862:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6564 00000100:00080000:1.0:1761234003.889864:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00080000:3.0:1761234003.889866:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761234003.889873:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895191616:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:3.0:1761234003.889874:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234003.889887:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234003.889893:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895191680:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234003.889916:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002625ec0b x1846780895191936/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761234003.889944:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234003.889951:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234003/real 1761234003] req@000000007aee79ba x1846780895191680/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234028 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234003.889969:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895191680/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234028 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234003.889976:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234003.889978:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234003.889981:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895191680:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761234003.889987:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895191744:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761234003.889998:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234003.890004:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234003/real 1761234003] req@0000000057e55ffa x1846780895191744/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234028 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234003.890014:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895191744/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234028 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234003.890025:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234003.890027:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234003.890029:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895191744:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761234003.890035:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895191808:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761234003.890056:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234003.890065:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234003/real 1761234003] req@000000002a0ba65f x1846780895191808/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234028 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234003.890076:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895191808/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234028 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234003.890084:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234003.890086:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234003.890089:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895191808:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761234003.890112:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895191872:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761234003.890134:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234003.890142:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234003/real 1761234003] req@00000000c9993cd4 x1846780895191872/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234028 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234003.890153:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895191872/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234028 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234003.890160:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234003.890163:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234003.890166:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895191872:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761234003.890172:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895191936:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761234003.890192:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234003.890229:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234003/real 1761234003] req@000000002625ec0b x1846780895191936/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234028 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234003.890241:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895191936/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234028 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234003.890248:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234003.890251:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234003.890254:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895191936:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761234004.657426:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234005.681403:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234006.705477:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234007.729375:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234008.753419:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761234009.009319:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234009.009330:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761234009.009334:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234009.009344:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6569 00000100:00080000:2.0:1761234009.009347:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234009.009353:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761234009.009387:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895192000/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234009.009407:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234009.009411:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234009.009412:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234009.009414:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6569 00000100:00080000:2.0:1761234009.009416:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234009.009418:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234009.009426:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234009.009433:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895192064/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234009.009438:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234009.009441:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234009.009442:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234009.009445:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6569 00000100:00080000:2.0:1761234009.009446:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234009.009448:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234009.009451:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234009.009458:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895192000:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761234009.009459:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895192128/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234009.009469:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234009.009471:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234009.009472:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234009.009474:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6569 00000100:00080000:2.0:1761234009.009475:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234009.009478:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234009.009481:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234009.009488:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895192192/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234009.009500:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234009.009503:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234009.009504:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234009.009509:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6569 00000100:00080000:2.0:1761234009.009511:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234009.009514:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234009.009517:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234009.009523:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895192256/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234009.009528:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234009.009531:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234009.009531:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234009.009534:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6569 00000100:00080000:2.0:1761234009.009535:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234009.009537:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234009.009540:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234009.009545:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895192320/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234009.009550:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234009.009553:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1761234009.009554:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895192064:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:2.0:1761234009.009555:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234009.009561:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6569 00000100:00080000:2.0:1761234009.009563:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234009.009566:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234009.009570:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234009.009578:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895192384/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761234009.009593:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895192128:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234009.009620:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895192192:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234009.009641:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895192256:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761234009.009654:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895192320:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234009.009664:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895192384:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761234009.777529:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234010.801322:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234011.825365:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1761234012.081490:0:2210:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:0.0:1761234012.081537:0:2210:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1761234012.081552:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761234012.081599:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761234012.081608:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761234012.081615:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761234012.081622:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761234012.081630:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761234012.081638:0:2210:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:1.0:1761234012.081668:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234009/real 1761234012] req@0000000040d5f361 x1846780895192128/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234034 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234012.081699:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895192128/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234034 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234012.081710:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234012.081713:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234012.081719:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895192128:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234012.081725:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234009/real 1761234012] req@000000009d315fa5 x1846780895192192/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234034 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234012.081735:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895192192/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234034 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234012.081742:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234012.081780:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234012.081784:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895192192:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234012.081791:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234009/real 1761234012] req@0000000033765045 x1846780895192256/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234034 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234012.081803:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895192256/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234034 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234012.081811:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234012.081814:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234012.081817:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895192256:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761234012.081822:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234009/real 1761234012] req@0000000064a2cea0 x1846780895192320/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234034 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234012.081839:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895192320/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234034 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234012.081846:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234012.081848:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234012.081851:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895192320:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234012.081857:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234009/real 1761234012] req@000000007091f547 x1846780895192384/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234034 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234012.081866:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895192384/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234034 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234012.081874:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234012.081876:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234012.081879:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895192384:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234012.081884:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234009/real 1761234012] req@000000004f29ca26 x1846780895192000/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234034 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234012.081894:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895192000/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234034 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234012.081901:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234012.081903:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234012.081906:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895192000:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761234012.081912:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234009/real 1761234012] req@000000007dcf84ae x1846780895192064/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234034 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234012.081927:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895192064/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234034 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234012.081934:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234012.081936:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234012.081939:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895192064:192.168.201.157@tcp:38:kworker/u8:2.0 00010000:00010000:3.0:1761234012.849404:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234013.873371:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761234014.129390:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761234014.129399:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761234014.129402:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761234014.129413:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6575 00000100:00080000:0.0:1761234014.129416:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761234014.129421:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761234014.129449:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c6e06344 x1846780895192448/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761234014.129472:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761234014.129476:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761234014.129478:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761234014.129482:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6575 00000100:00080000:0.0:1761234014.129484:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761234014.129489:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761234014.129498:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761234014.129510:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005593c994 x1846780895192512/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234014.129512:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895192448:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761234014.129519:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761234014.129522:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761234014.129523:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761234014.129527:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6575 00000100:00080000:0.0:1761234014.129528:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761234014.129531:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761234014.129535:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761234014.129542:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001f9ec359 x1846780895192576/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761234014.129549:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761234014.129552:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761234014.129553:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761234014.129556:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6575 00000100:00080000:0.0:1761234014.129558:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761234014.129561:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761234014.129564:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761234014.129571:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000efa29eb9 x1846780895192640/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761234014.129577:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761234014.129580:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761234014.129582:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1761234014.129582:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761234014.129585:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6575 00000100:00080000:0.0:1761234014.129586:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761234014.129589:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761234014.129593:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761234014.129600:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d61dd06 x1846780895192704/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234014.129602:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234014/real 1761234014] req@00000000c6e06344 x1846780895192448/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234039 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761234014.129608:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761234014.129611:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761234014.129612:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761234014.129615:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6575 00000100:00080000:0.0:1761234014.129617:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761234014.129617:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c6e06344 x1846780895192448/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234039 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:0.0:1761234014.129620:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761234014.129624:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761234014.129627:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234014.129630:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761234014.129631:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ef655201 x1846780895192768/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234014.129634:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c6e06344 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895192448:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:0.0:1761234014.129639:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761234014.129641:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761234014.129643:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761234014.129645:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6575 00000100:00080000:0.0:1761234014.129647:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761234014.129649:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761234014.129652:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234014.129656:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895192512:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:0.0:1761234014.129659:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004130b6bd x1846780895192832/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761234014.129677:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234014.129686:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234014/real 1761234014] req@000000005593c994 x1846780895192512/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234039 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234014.129697:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005593c994 x1846780895192512/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234039 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234014.129729:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234014.129732:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234014.129734:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005593c994 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895192512:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761234014.129741:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895192576:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761234014.129764:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234014.129772:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234014/real 1761234014] req@000000001f9ec359 x1846780895192576/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234039 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234014.129783:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001f9ec359 x1846780895192576/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234039 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234014.129789:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234014.129792:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234014.129795:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001f9ec359 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895192576:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761234014.129800:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895192640:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761234014.129816:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234014.129824:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234014/real 1761234014] req@00000000efa29eb9 x1846780895192640/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234039 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234014.129834:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000efa29eb9 x1846780895192640/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234039 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234014.129840:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234014.129842:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234014.129844:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000efa29eb9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895192640:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761234014.129850:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895192704:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761234014.129869:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234014.129876:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234014/real 1761234014] req@000000006d61dd06 x1846780895192704/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234039 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234014.129887:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d61dd06 x1846780895192704/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234039 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234014.129895:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234014.129897:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234014.129899:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d61dd06 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895192704:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761234014.129905:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895192768:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761234014.129924:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234014.129931:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234014/real 1761234014] req@00000000ef655201 x1846780895192768/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234039 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234014.129941:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ef655201 x1846780895192768/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234039 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234014.129947:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234014.129949:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234014.129951:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ef655201 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895192768:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761234014.129975:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895192832:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761234014.130013:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234014.130022:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234014/real 1761234014] req@000000004130b6bd x1846780895192832/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234039 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234014.130033:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004130b6bd x1846780895192832/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234039 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234014.130040:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234014.130042:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234014.130044:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004130b6bd pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895192832:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761234014.897424:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234015.921331:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234016.945522:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234017.969372:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234018.993421:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761234019.249328:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234019.249339:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761234019.249343:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234019.249354:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6580 00000100:00080000:3.0:1761234019.249358:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234019.249365:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761234019.249400:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b40830d x1846780895192896/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234019.249427:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234019.249431:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234019.249433:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234019.249436:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6580 00000100:00080000:3.0:1761234019.249438:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234019.249442:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234019.249451:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234019.249462:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000829a5595 x1846780895192960/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234019.249470:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234019.249474:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234019.249475:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234019.249478:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6580 00000100:00080000:3.0:1761234019.249480:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234019.249483:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234019.249488:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234019.249497:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000089b8c4f1 x1846780895193024/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234019.249507:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761234019.249511:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895192896:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:3.0:1761234019.249511:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234019.249513:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234019.249521:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6580 00000100:00080000:3.0:1761234019.249523:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234019.249527:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234019.249531:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234019.249541:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000070fa3bd6 x1846780895193088/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234019.249549:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234019.249553:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234019.249554:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234019.249558:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6580 00000100:00080000:3.0:1761234019.249560:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234019.249564:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234019.249568:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234019.249580:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090702bc4 x1846780895193152/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761234019.249581:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761234019.249591:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234019.249595:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234019.249596:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234019.249601:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6580 00000100:00100000:1.0:1761234019.249603:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234019/real 1761234019] req@000000003b40830d x1846780895192896/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234044 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234019.249604:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234019.249624:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234019.249632:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234019.249637:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895192896/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234044 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234019.249644:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6fdd7f9 x1846780895193216/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234019.249647:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234019.249651:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234019.249658:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895192896:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:3.0:1761234019.249660:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234019.249664:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1761234019.249665:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895192960:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:3.0:1761234019.249668:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234019.249674:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6580 00000100:00080000:3.0:1761234019.249677:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234019.249680:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:1.0:1761234019.249685:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:3.0:1761234019.249685:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234019.249692:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234019/real 1761234019] req@00000000829a5595 x1846780895192960/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234044 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234019.249700:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008e58c4b8 x1846780895193280/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761234019.249705:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895192960/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234044 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234019.249727:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234019.249729:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234019.249731:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895192960:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761234019.249752:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895193024:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761234019.249764:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234019.249768:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234019/real 1761234019] req@0000000089b8c4f1 x1846780895193024/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234044 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234019.249776:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895193024/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234044 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234019.249783:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234019.249786:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234019.249788:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895193024:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761234019.249794:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895193088:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761234019.249812:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234019.249820:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234019/real 1761234019] req@0000000070fa3bd6 x1846780895193088/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234044 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234019.249831:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895193088/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234044 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234019.249838:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234019.249841:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234019.249844:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895193088:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761234019.249850:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895193152:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761234019.249867:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234019.249873:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234019/real 1761234019] req@0000000090702bc4 x1846780895193152/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234044 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234019.249880:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895193152/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234044 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234019.249885:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234019.249886:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234019.249888:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895193152:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761234019.249891:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895193216:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761234019.249901:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234019.249905:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234019/real 1761234019] req@00000000b6fdd7f9 x1846780895193216/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234044 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234019.249911:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895193216/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234044 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234019.249915:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234019.249916:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234019.249918:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895193216:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761234019.249921:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895193280:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761234019.249929:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234019.249933:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234019/real 1761234019] req@000000008e58c4b8 x1846780895193280/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234044 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234019.249939:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895193280/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234044 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234019.249944:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234019.249945:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234019.249946:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895193280:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761234020.017377:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234021.041467:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234022.065501:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234023.089476:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234024.113479:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761234024.369310:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234024.369323:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761234024.369328:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234024.369341:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6585 00000100:00080000:3.0:1761234024.369345:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234024.369352:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761234024.369396:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e99e5a12 x1846780895193344/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234024.369422:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234024.369426:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234024.369427:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234024.369431:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6585 00000100:00080000:3.0:1761234024.369433:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234024.369437:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234024.369448:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234024.369459:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007adf99ab x1846780895193408/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234024.369466:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234024.369469:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234024.369471:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234024.369474:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6585 00000100:00080000:3.0:1761234024.369476:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234024.369480:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234024.369486:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234024.369499:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000080131ea7 x1846780895193472/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761234024.369505:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895193344:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:3.0:1761234024.369523:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234024.369527:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234024.369529:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234024.369533:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6585 00000100:00080000:3.0:1761234024.369535:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234024.369539:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234024.369544:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234024.369556:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008e58c4b8 x1846780895193536/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234024.369563:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234024.369566:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234024.369567:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234024.369571:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6585 00000100:00080000:3.0:1761234024.369573:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234024.369576:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234024.369580:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234024.369587:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6fdd7f9 x1846780895193600/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234024.369594:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234024.369597:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234024.369598:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234024.369601:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6585 00000100:00080000:3.0:1761234024.369604:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234024.369607:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234024.369610:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761234024.369614:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234024.369617:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090702bc4 x1846780895193664/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234024.369624:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234024.369627:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234024.369628:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234024.369631:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6585 00000100:00080000:3.0:1761234024.369633:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234024.369636:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234024.369639:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234024.369642:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234024/real 1761234024] req@00000000e99e5a12 x1846780895193344/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234049 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234024.369646:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000070fa3bd6 x1846780895193728/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761234024.369660:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895193344/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234049 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234024.369670:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234024.369674:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234024.369678:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895193344:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761234024.369685:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895193408:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761234024.369708:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234024.369716:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234024/real 1761234024] req@000000007adf99ab x1846780895193408/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234049 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234024.369723:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895193408/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234049 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234024.369733:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234024.369734:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234024.369736:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895193408:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761234024.369758:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895193472:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234024.369771:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234024.369776:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234024/real 1761234024] req@0000000080131ea7 x1846780895193472/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234049 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234024.369782:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895193472/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234049 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234024.369787:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234024.369788:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234024.369789:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895193472:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234024.369793:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895193536:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234024.369802:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234024.369807:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234024/real 1761234024] req@000000008e58c4b8 x1846780895193536/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234049 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234024.369813:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895193536/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234049 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234024.369818:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234024.369819:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234024.369820:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895193536:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234024.369824:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895193600:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761234024.369834:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234024.369838:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234024/real 1761234024] req@00000000b6fdd7f9 x1846780895193600/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234049 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234024.369845:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895193600/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234049 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234024.369849:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234024.369851:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234024.369852:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895193600:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761234024.369855:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895193664:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234024.369864:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234024.369868:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234024/real 1761234024] req@0000000090702bc4 x1846780895193664/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234049 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234024.369873:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895193664/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234049 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234024.369877:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234024.369878:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234024.369879:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895193664:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234024.369882:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895193728:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234024.369891:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234024.369895:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234024/real 1761234024] req@0000000070fa3bd6 x1846780895193728/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234049 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234024.369902:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895193728/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234049 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234024.369906:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234024.369907:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234024.369909:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895193728:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761234025.137466:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234026.161414:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234027.185479:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234028.209419:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234029.233527:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761234029.489363:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234029.489377:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761234029.489382:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234029.489396:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6590 00000100:00080000:3.0:1761234029.489400:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234029.489409:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761234029.489504:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000089b8c4f1 x1846780895193792/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234029.489536:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234029.489541:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234029.489543:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234029.489547:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6590 00000100:00080000:3.0:1761234029.489549:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234029.489552:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234029.489563:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234029.489575:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000829a5595 x1846780895193856/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234029.489582:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234029.489585:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234029.489587:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234029.489593:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6590 00000100:00080000:3.0:1761234029.489595:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234029.489597:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234029.489600:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234029.489607:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b40830d x1846780895193920/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234029.489615:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234029.489620:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234029.489622:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234029.489628:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6590 00000100:00080000:3.0:1761234029.489633:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761234029.489634:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895193792:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:3.0:1761234029.489642:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234029.489651:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234029.489670:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002625ec0b x1846780895193984/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234029.489681:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234029.489685:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234029.489688:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234029.489692:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6590 00000100:00080000:3.0:1761234029.489695:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234029.489699:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234029.489705:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234029.489724:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c9993cd4 x1846780895194048/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761234029.489725:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761234029.489740:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234029.489759:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234029.489762:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761234029.489763:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234029/real 1761234029] req@0000000089b8c4f1 x1846780895193792/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234054 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234029.489779:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6590 00000100:00080000:3.0:1761234029.489785:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761234029.489795:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895193792/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234054 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234029.489795:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234029.489809:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1761234029.489814:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234029.489819:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234029.489824:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895193792:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761234029.489830:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a0ba65f x1846780895194112/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761234029.489835:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895193856:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:3.0:1761234029.489858:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234029.489864:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1761234029.489868:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761234029.489870:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761234029.489878:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234029/real 1761234029] req@00000000829a5595 x1846780895193856/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234054 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234029.489883:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6590 00000100:00080000:3.0:1761234029.489892:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761234029.489903:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895193856/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234054 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234029.489907:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234029.489927:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1761234029.489931:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234029.489934:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234029.489939:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895193856:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761234029.489955:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e55ffa x1846780895194176/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761234029.489973:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895193920:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234029.489994:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234029.490003:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234029/real 1761234029] req@000000003b40830d x1846780895193920/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234054 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234029.490015:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895193920/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234054 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234029.490020:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234029.490023:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234029.490025:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895193920:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234029.490030:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895193984:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234029.490042:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234029.490047:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234029/real 1761234029] req@000000002625ec0b x1846780895193984/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234054 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234029.490054:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895193984/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234054 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234029.490060:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234029.490061:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234029.490063:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895193984:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234029.490068:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895194048:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761234029.490081:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234029.490085:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234029/real 1761234029] req@00000000c9993cd4 x1846780895194048/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234054 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234029.490093:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895194048/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234054 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234029.490098:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234029.490100:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234029.490102:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895194048:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761234029.490106:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895194112:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234029.490117:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234029.490121:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234029/real 1761234029] req@000000002a0ba65f x1846780895194112/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234054 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234029.490128:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895194112/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234054 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234029.490133:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234029.490135:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234029.490136:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895194112:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234029.490150:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895194176:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234029.490162:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234029.490167:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234029/real 1761234029] req@0000000057e55ffa x1846780895194176/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234054 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234029.490175:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895194176/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234054 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234029.490180:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234029.490182:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234029.490184:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895194176:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761234030.257381:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234031.281413:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234032.305409:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234033.329403:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234034.353396:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761234034.609422:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234034.609436:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761234034.609440:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234034.609452:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6595 00000100:00080000:3.0:1761234034.609456:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234034.609464:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761234034.609515:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007aee79ba x1846780895194240/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234034.609549:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234034.609556:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234034.609558:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234034.609564:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6595 00000100:00080000:3.0:1761234034.609568:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234034.609577:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234034.609588:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234034.609590:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895194240:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761234034.609607:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009706a782 x1846780895194304/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234034.609619:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234034.609623:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234034.609624:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234034.609628:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6595 00000100:00080000:3.0:1761234034.609631:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234034.609634:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234034.609638:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234034.609648:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000bbf6af7b x1846780895194368/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234034.609656:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234034.609659:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234034.609660:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234034.609663:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6595 00000100:00080000:3.0:1761234034.609665:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1761234034.609670:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761234034.609674:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234034.609679:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234034.609691:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e55ffa x1846780895194432/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234034.609700:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234034.609704:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234034.609706:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761234034.609712:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234034/real 1761234034] req@000000007aee79ba x1846780895194240/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234059 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234034.609713:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6595 00000100:00080000:3.0:1761234034.609718:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234034.609725:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234034.609733:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234034.609747:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895194240/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234059 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234034.609748:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a0ba65f x1846780895194496/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234034.609789:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234034.609792:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234034.609794:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234034.609797:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234034.609800:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:3.0:1761234034.609804:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6595 00000100:00080000:3.0:1761234034.609808:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761234034.609811:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895194240:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:3.0:1761234034.609816:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234034.609823:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234034.609833:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c9993cd4 x1846780895194560/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234034.609840:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234034.609843:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234034.609845:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761234034.609848:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895194304:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:3.0:1761234034.609854:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6595 00000100:00080000:3.0:1761234034.609859:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234034.609862:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234034.609866:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234034.609877:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002625ec0b x1846780895194624/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761234034.609885:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234034.609903:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234034/real 1761234034] req@000000009706a782 x1846780895194304/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234059 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234034.609916:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895194304/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234059 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234034.609931:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234034.609933:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234034.609937:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895194304:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761234034.609942:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895194368:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234034.609958:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234034.609964:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234034/real 1761234034] req@00000000bbf6af7b x1846780895194368/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234059 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234034.609974:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895194368/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234059 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234034.609980:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234034.609982:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234034.609984:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895194368:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234034.609989:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895194432:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234034.610002:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234034.610008:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234034/real 1761234034] req@0000000057e55ffa x1846780895194432/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234059 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234034.610017:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895194432/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234059 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234034.610023:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234034.610025:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234034.610027:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895194432:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234034.610033:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895194496:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761234034.610046:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234034.610052:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234034/real 1761234034] req@000000002a0ba65f x1846780895194496/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234059 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234034.610061:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895194496/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234059 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234034.610068:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234034.610070:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234034.610074:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895194496:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761234034.610081:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895194560:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234034.610116:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234034.610129:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234034/real 1761234034] req@00000000c9993cd4 x1846780895194560/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234059 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234034.610144:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895194560/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234059 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234034.610155:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234034.610158:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234034.610162:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895194560:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234034.610193:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895194624:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234034.610286:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234034.610296:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234034/real 1761234034] req@000000002625ec0b x1846780895194624/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234059 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234034.610307:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895194624/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234059 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234034.610314:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234034.610335:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234034.610340:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895194624:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761234035.377338:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234036.401376:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234037.425405:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234038.449420:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234039.473376:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761234039.729305:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234039.729314:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761234039.729317:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234039.729326:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6600 00000100:00080000:3.0:1761234039.729329:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234039.729336:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761234039.729367:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b40830d x1846780895194688/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234039.729390:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234039.729392:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234039.729393:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234039.729396:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6600 00000100:00080000:3.0:1761234039.729397:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234039.729399:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234039.729405:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234039.729412:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000829a5595 x1846780895194752/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234039.729418:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234039.729420:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234039.729421:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234039.729424:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6600 00000100:00080000:3.0:1761234039.729427:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234039.729429:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234039.729433:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234039.729438:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000089b8c4f1 x1846780895194816/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234039.729443:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234039.729445:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234039.729446:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761234039.729450:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895194688:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:3.0:1761234039.729452:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6600 00000100:00080000:3.0:1761234039.729456:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234039.729460:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234039.729462:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234039.729477:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000070fa3bd6 x1846780895194880/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234039.729482:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234039.729484:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234039.729485:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234039.729487:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6600 00000100:00080000:3.0:1761234039.729489:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234039.729505:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234039.729508:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234039.729517:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090702bc4 x1846780895194944/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234039.729522:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234039.729525:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234039.729526:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234039.729528:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6600 00000100:00080000:3.0:1761234039.729530:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234039.729532:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234039.729534:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234039.729540:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6fdd7f9 x1846780895195008/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234039.729546:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1761234039.729548:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761234039.729554:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234039.729555:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234039.729558:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6600 00000100:00080000:3.0:1761234039.729560:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234039.729574:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234039.729576:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234039.729583:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008e58c4b8 x1846780895195072/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761234039.729588:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234039/real 1761234039] req@000000003b40830d x1846780895194688/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234064 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234039.729608:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895194688/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234064 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234039.729618:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234039.729621:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234039.729625:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895194688:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761234039.729632:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895194752:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761234039.729653:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234039.729661:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234039/real 1761234039] req@00000000829a5595 x1846780895194752/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234064 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234039.729671:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895194752/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234064 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234039.729684:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234039.729686:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234039.729689:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895194752:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761234039.729713:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895194816:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234039.729731:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234039.729738:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234039/real 1761234039] req@0000000089b8c4f1 x1846780895194816/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234064 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234039.729748:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895194816/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234064 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234039.729755:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234039.729757:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234039.729760:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895194816:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234039.729765:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895194880:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234039.729781:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234039.729787:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234039/real 1761234039] req@0000000070fa3bd6 x1846780895194880/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234064 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234039.729797:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895194880/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234064 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234039.729803:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234039.729805:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234039.729808:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895194880:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234039.729813:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895194944:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761234039.729830:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234039.729836:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234039/real 1761234039] req@0000000090702bc4 x1846780895194944/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234064 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234039.729845:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895194944/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234064 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234039.729852:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234039.729854:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234039.729857:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895194944:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761234039.729863:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895195008:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234039.729878:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234039.729884:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234039/real 1761234039] req@00000000b6fdd7f9 x1846780895195008/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234064 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234039.729894:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895195008/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234064 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234039.729901:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234039.729903:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234039.729906:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895195008:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234039.729911:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895195072:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234039.729926:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234039.729933:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234039/real 1761234039] req@000000008e58c4b8 x1846780895195072/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234064 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234039.729943:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895195072/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234064 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234039.729951:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234039.729953:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234039.729956:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895195072:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761234040.497404:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234041.521435:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234042.545437:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234043.569401:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234044.593423:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761234044.849306:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234044.849319:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761234044.849323:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234044.849334:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6605 00000100:00080000:2.0:1761234044.849339:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234044.849346:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761234044.849377:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895195136/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234044.849400:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234044.849403:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234044.849404:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234044.849407:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6605 00000100:00080000:2.0:1761234044.849408:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234044.849411:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234044.849418:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234044.849427:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895195200/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234044.849433:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234044.849436:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234044.849439:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234044.849442:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6605 00000100:00080000:2.0:1761234044.849443:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234044.849445:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234044.849448:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234044.849457:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895195136:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761234044.849458:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895195264/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234044.849464:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234044.849465:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234044.849466:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234044.849472:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6605 00000100:00080000:2.0:1761234044.849474:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234044.849476:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234044.849479:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234044.849483:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895195328/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234044.849487:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234044.849489:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234044.849490:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234044.849493:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6605 00000100:00080000:2.0:1761234044.849494:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234044.849497:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234044.849499:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234044.849505:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895195392/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234044.849510:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234044.849513:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234044.849514:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234044.849515:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6605 00000100:00080000:2.0:1761234044.849516:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234044.849518:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234044.849521:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234044.849528:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895195456/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234044.849533:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234044.849534:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234044.849535:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234044.849537:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6605 00000100:00080000:2.0:1761234044.849538:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234044.849542:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:1.0:1761234044.849545:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:2.0:1761234044.849547:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234044.849554:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895195520/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761234044.849568:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234044/real 1761234044] req@00000000fd3935a6 x1846780895195136/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234069 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234044.849585:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895195136/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234069 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234044.849595:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234044.849597:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234044.849602:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895195136:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761234044.849609:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895195200:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761234044.849633:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234044.849641:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234044/real 1761234044] req@00000000135e9a6e x1846780895195200/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234069 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234044.849648:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895195200/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234069 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234044.849657:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234044.849659:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234044.849661:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895195200:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761234044.849681:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895195264:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234044.849694:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234044.849699:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234044/real 1761234044] req@000000006ff1fdbc x1846780895195264/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234069 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234044.849705:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895195264/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234069 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234044.849710:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234044.849711:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234044.849713:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895195264:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234044.849716:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895195328:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234044.849726:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234044.849729:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234044/real 1761234044] req@0000000088939481 x1846780895195328/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234069 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234044.849736:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895195328/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234069 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234044.849740:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234044.849741:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234044.849742:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895195328:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234044.849746:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895195392:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761234044.849755:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234044.849759:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234044/real 1761234044] req@000000006d8c3eb0 x1846780895195392/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234069 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234044.849765:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895195392/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234069 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234044.849769:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234044.849770:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234044.849772:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895195392:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761234044.849775:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895195456:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234044.849783:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234044.849787:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234044/real 1761234044] req@000000006c1041ba x1846780895195456/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234069 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234044.849792:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895195456/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234069 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234044.849796:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234044.849798:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234044.849799:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895195456:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234044.849803:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895195520:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234044.849811:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234044.849815:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234044/real 1761234044] req@00000000fd4aa3a2 x1846780895195520/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234069 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234044.849821:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895195520/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234069 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234044.849825:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234044.849826:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234044.849828:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895195520:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761234045.617438:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234046.641370:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234047.665398:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234048.689357:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234049.713476:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761234049.969377:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234049.969385:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761234049.969389:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234049.969401:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6610 00000100:00080000:3.0:1761234049.969405:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234049.969411:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761234049.969450:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000080131ea7 x1846780895195584/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234049.969480:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234049.969484:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234049.969486:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234049.969490:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6610 00000100:00080000:3.0:1761234049.969492:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234049.969496:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234049.969506:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234049.969518:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007adf99ab x1846780895195648/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234049.969526:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234049.969530:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234049.969532:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234049.969536:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6610 00000100:00080000:3.0:1761234049.969538:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234049.969542:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234049.969547:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234049.969550:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895195584:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761234049.969559:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e99e5a12 x1846780895195712/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234049.969569:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234049.969573:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234049.969575:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234049.969578:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6610 00000100:00080000:3.0:1761234049.969581:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234049.969584:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234049.969589:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234049.969601:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008e58c4b8 x1846780895195776/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234049.969610:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234049.969613:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234049.969615:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1761234049.969620:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761234049.969623:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6610 00000100:00080000:3.0:1761234049.969627:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234049.969631:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234049.969636:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234049.969642:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234049/real 1761234049] req@0000000080131ea7 x1846780895195584/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234074 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234049.969649:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6fdd7f9 x1846780895195840/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761234049.969660:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895195584/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234074 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234049.969661:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234049.969669:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234049.969671:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234049.969676:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234049.969678:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6610 00000100:00080000:1.0:1761234049.969680:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:3.0:1761234049.969683:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761234049.969686:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895195584:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:3.0:1761234049.969691:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761234049.969696:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895195648:192.168.201.157@tcp:38:kworker/u8:2.0 00000008:00100000:3.0:1761234049.969698:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234049.969714:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090702bc4 x1846780895195904/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761234049.969719:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234049.969727:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234049/real 1761234049] req@000000007adf99ab x1846780895195648/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234074 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234049.969728:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234049.969737:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234049.969741:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761234049.969744:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895195648/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234074 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234049.969749:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6610 00000100:00080000:3.0:1761234049.969753:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234049.969759:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:1.0:1761234049.969767:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1761234049.969769:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1761234049.969770:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234049.969776:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895195648:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761234049.969785:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000070fa3bd6 x1846780895195968/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761234049.969797:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895195712:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234049.969812:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234049.969819:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234049/real 1761234049] req@00000000e99e5a12 x1846780895195712/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234074 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234049.969826:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895195712/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234074 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234049.969831:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234049.969832:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234049.969834:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895195712:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234049.969837:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895195776:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234049.969846:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234049.969850:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234049/real 1761234049] req@000000008e58c4b8 x1846780895195776/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234074 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234049.969856:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895195776/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234074 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234049.969860:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234049.969861:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234049.969862:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895195776:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234049.969866:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895195840:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761234049.969875:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234049.969879:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234049/real 1761234049] req@00000000b6fdd7f9 x1846780895195840/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234074 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234049.969890:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895195840/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234074 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234049.969895:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234049.969896:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234049.969898:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895195840:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761234049.969901:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895195904:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234049.969910:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234049.969914:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234049/real 1761234049] req@0000000090702bc4 x1846780895195904/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234074 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234049.969920:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895195904/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234074 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234049.969924:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234049.969925:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234049.969927:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895195904:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234049.969938:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895195968:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234049.969947:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234049.969950:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234049/real 1761234049] req@0000000070fa3bd6 x1846780895195968/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234074 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234049.969957:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895195968/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234074 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234049.969961:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234049.969962:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234049.969964:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895195968:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761234050.737402:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234051.761523:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234052.785404:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234053.809421:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234054.833363:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761234055.089297:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234055.089305:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761234055.089309:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234055.089319:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6616 00000100:00080000:3.0:1761234055.089322:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234055.089329:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761234055.089361:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000089b8c4f1 x1846780895196032/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234055.089384:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234055.089386:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234055.089388:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234055.089391:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6616 00000100:00080000:3.0:1761234055.089393:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234055.089396:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234055.089404:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234055.089412:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000829a5595 x1846780895196096/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234055.089419:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234055.089421:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234055.089422:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234055.089424:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6616 00000100:00080000:3.0:1761234055.089425:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234055.089427:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234055.089430:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234055.089436:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b40830d x1846780895196160/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234055.089441:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234055.089443:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234055.089444:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234055.089447:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6616 00000100:00080000:3.0:1761234055.089449:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234055.089451:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234055.089453:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234055.089458:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002625ec0b x1846780895196224/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234055.089468:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761234055.089469:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895196032:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:3.0:1761234055.089471:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234055.089473:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234055.089478:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6616 00000100:00080000:3.0:1761234055.089480:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234055.089482:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234055.089485:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234055.089493:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c9993cd4 x1846780895196288/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234055.089501:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234055.089504:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234055.089506:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234055.089510:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6616 00000100:00080000:3.0:1761234055.089512:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234055.089516:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234055.089520:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234055.089532:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a0ba65f x1846780895196352/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:1.0:1761234055.089536:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761234055.089545:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234055.089549:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234055.089551:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234055.089556:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6616 00000100:00100000:1.0:1761234055.089559:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234055/real 1761234055] req@0000000089b8c4f1 x1846780895196032/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234080 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234055.089559:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234055.089567:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234055.089573:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234055.089580:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895196032/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234080 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234055.089586:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e55ffa x1846780895196416/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234055.089594:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234055.089598:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234055.089602:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895196032:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761234055.089608:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895196096:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761234055.089624:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234055.089628:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234055/real 1761234055] req@00000000829a5595 x1846780895196096/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234080 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234055.089635:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895196096/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234080 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234055.089644:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234055.089646:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234055.089647:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895196096:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761234055.089650:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895196160:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234055.089661:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234055.089664:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234055/real 1761234055] req@000000003b40830d x1846780895196160/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234080 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234055.089670:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895196160/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234080 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234055.089674:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234055.089675:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234055.089677:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895196160:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234055.089696:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895196224:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234055.089706:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234055.089710:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234055/real 1761234055] req@000000002625ec0b x1846780895196224/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234080 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234055.089716:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895196224/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234080 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234055.089720:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234055.089721:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234055.089723:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895196224:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234055.089727:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895196288:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761234055.089736:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234055.089740:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234055/real 1761234055] req@00000000c9993cd4 x1846780895196288/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234080 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234055.089747:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895196288/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234080 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234055.089751:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234055.089752:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234055.089753:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895196288:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761234055.089756:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895196352:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234055.089765:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234055.089769:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234055/real 1761234055] req@000000002a0ba65f x1846780895196352/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234080 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234055.089774:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895196352/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234080 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234055.089779:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234055.089780:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234055.089781:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895196352:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234055.089784:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895196416:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234055.089793:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234055.089797:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234055/real 1761234055] req@0000000057e55ffa x1846780895196416/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234080 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234055.089803:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895196416/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234080 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234055.089808:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234055.089809:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234055.089810:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895196416:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761234055.857374:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234056.881406:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234057.905455:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234058.929362:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234059.953402:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761234060.209334:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234060.209342:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761234060.209345:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234060.209354:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6621 00000100:00080000:3.0:1761234060.209358:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234060.209364:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761234060.209400:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000bbf6af7b x1846780895196480/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234060.209415:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234060.209418:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234060.209419:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234060.209422:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6621 00000100:00080000:3.0:1761234060.209424:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234060.209426:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761234060.209433:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895196480:192.168.201.157@tcp:250:kworker/u8:2.0 00000008:00100000:3.0:1761234060.209434:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234060.209443:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009706a782 x1846780895196544/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234060.209449:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234060.209452:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234060.209453:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234060.209456:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6621 00000100:00080000:3.0:1761234060.209457:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234060.209459:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234060.209462:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234060.209468:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007aee79ba x1846780895196608/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234060.209473:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234060.209475:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234060.209476:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234060.209479:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6621 00000100:00080000:3.0:1761234060.209480:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234060.209482:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234060.209485:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234060.209491:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e55ffa x1846780895196672/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234060.209496:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234060.209498:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234060.209499:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234060.209501:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6621 00000100:00080000:3.0:1761234060.209502:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234060.209504:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234060.209507:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761234060.209509:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234060.209517:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a0ba65f x1846780895196736/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234060.209524:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234060.209526:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234060.209527:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234060.209530:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6621 00000100:00080000:3.0:1761234060.209531:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1761234060.209533:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234060/real 1761234060] req@00000000bbf6af7b x1846780895196480/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234085 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234060.209536:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234060.209541:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234060.209550:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c9993cd4 x1846780895196800/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761234060.209556:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895196480/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234085 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234060.209559:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234060.209565:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234060.209567:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234060.209573:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6621 00000100:00080000:3.0:1761234060.209574:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234060.209576:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234060.209578:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1761234060.209583:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1761234060.209585:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002625ec0b x1846780895196864/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234060.209590:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234060.209595:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895196480:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761234060.209618:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895196544:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761234060.209635:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234060.209640:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234060/real 1761234060] req@000000009706a782 x1846780895196544/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234085 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234060.209647:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895196544/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234085 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234060.209656:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234060.209658:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234060.209659:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895196544:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761234060.209663:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895196608:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234060.209673:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234060.209677:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234060/real 1761234060] req@000000007aee79ba x1846780895196608/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234085 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234060.209683:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895196608/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234085 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234060.209687:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234060.209688:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234060.209689:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895196608:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234060.209692:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895196672:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234060.209701:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234060.209705:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234060/real 1761234060] req@0000000057e55ffa x1846780895196672/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234085 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234060.209710:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895196672/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234085 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234060.209715:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234060.209716:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234060.209718:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895196672:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234060.209721:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895196736:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761234060.209731:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234060.209735:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234060/real 1761234060] req@000000002a0ba65f x1846780895196736/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234085 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234060.209740:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895196736/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234085 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234060.209744:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234060.209746:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234060.209747:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895196736:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761234060.209750:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895196800:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234060.209759:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234060.209762:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234060/real 1761234060] req@00000000c9993cd4 x1846780895196800/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234085 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234060.209768:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895196800/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234085 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234060.209783:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234060.209784:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234060.209786:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895196800:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234060.209789:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895196864:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234060.209798:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234060.209801:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234060/real 1761234060] req@000000002625ec0b x1846780895196864/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234085 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234060.209808:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895196864/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234085 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234060.209812:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234060.209813:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234060.209814:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895196864:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761234060.977465:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234062.001428:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234063.025481:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234064.049425:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234065.073488:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761234065.329454:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234065.329471:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761234065.329477:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234065.329508:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6626 00000100:00080000:1.0:1761234065.329514:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234065.329530:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761234065.329602:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000880f16b4 x1846780895196928/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234065.329638:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234065.329643:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761234065.329645:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234065.329650:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6626 00000100:00080000:1.0:1761234065.329652:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234065.329656:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234065.329669:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234065.329684:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000053e52331 x1846780895196992/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234065.329707:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234065.329712:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761234065.329714:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234065.329718:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6626 00000100:00080000:1.0:1761234065.329720:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234065.329724:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234065.329731:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234065.329762:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895196928:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761234065.329770:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090ac513e x1846780895197056/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234065.329799:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234065.329803:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761234065.329805:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234065.329809:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6626 00000100:00080000:1.0:1761234065.329812:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234065.329816:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234065.329821:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234065.329834:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d66930c2 x1846780895197120/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234065.329844:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234065.329848:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761234065.329850:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234065.329855:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6626 00000100:00080000:1.0:1761234065.329858:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234065.329864:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234065.329870:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234065.329880:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006a5e9865 x1846780895197184/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234065.329891:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234065.329894:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761234065.329896:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234065.329900:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6626 00000100:00080000:1.0:1761234065.329903:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234065.329914:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:3.0:1761234065.329921:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:1.0:1761234065.329924:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234065.329940:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004361f727 x1846780895197248/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234065.329951:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234065.329955:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761234065.329958:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234065.329963:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6626 00000100:00080000:1.0:1761234065.329966:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234065.329971:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234065.329978:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234065.329979:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234065/real 1761234065] req@00000000880f16b4 x1846780895196928/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234090 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234065.330004:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ffe86d04 x1846780895197312/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234065.330050:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000880f16b4 x1846780895196928/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234090 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234065.330072:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234065.330079:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234065.330087:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895196928:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761234065.330102:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895196992:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761234065.330176:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234065.330187:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234065/real 1761234065] req@0000000053e52331 x1846780895196992/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234090 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234065.330223:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000053e52331 x1846780895196992/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234090 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234065.330236:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234065.330239:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234065.330244:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895196992:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234065.330284:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895197056:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234065.330317:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234065.330329:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234065/real 1761234065] req@0000000090ac513e x1846780895197056/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234090 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234065.330346:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090ac513e x1846780895197056/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234090 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234065.330366:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234065.330369:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234065.330373:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895197056:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234065.330381:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895197120:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234065.330410:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234065.330423:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234065/real 1761234065] req@00000000d66930c2 x1846780895197120/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234090 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234065.330439:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d66930c2 x1846780895197120/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234090 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234065.330450:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234065.330454:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234065.330459:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895197120:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234065.330467:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a5e9865 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895197184:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761234065.330497:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234065.330511:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234065/real 1761234065] req@000000006a5e9865 x1846780895197184/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234090 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234065.330527:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006a5e9865 x1846780895197184/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234090 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234065.330536:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234065.330538:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234065.330541:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a5e9865 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895197184:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234065.330548:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004361f727 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895197248:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234065.330569:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234065.330576:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234065/real 1761234065] req@000000004361f727 x1846780895197248/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234090 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234065.330586:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004361f727 x1846780895197248/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234090 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234065.330592:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234065.330595:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234065.330597:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004361f727 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895197248:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234065.330601:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895197312:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234065.330615:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234065.330620:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234065/real 1761234065] req@00000000ffe86d04 x1846780895197312/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234090 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234065.330649:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ffe86d04 x1846780895197312/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234090 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234065.330658:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234065.330660:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234065.330663:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895197312:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761234066.097506:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234067.121558:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234068.145466:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234069.169479:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234070.193409:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761234070.449439:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234070.449454:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761234070.449461:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234070.449476:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6631 00000100:00080000:3.0:1761234070.449480:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234070.449490:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761234070.449545:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b40830d x1846780895197376/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234070.449581:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234070.449585:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234070.449586:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234070.449590:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6631 00000100:00080000:3.0:1761234070.449593:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234070.449596:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234070.449608:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234070.449620:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000829a5595 x1846780895197440/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234070.449631:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234070.449634:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1761234070.449653:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895197376:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:3.0:1761234070.449661:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234070.449668:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6631 00000100:00080000:3.0:1761234070.449671:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234070.449674:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234070.449680:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234070.449696:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000089b8c4f1 x1846780895197504/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234070.449705:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234070.449710:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234070.449711:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234070.449715:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6631 00000100:00080000:3.0:1761234070.449717:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234070.449720:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234070.449724:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234070.449734:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000070fa3bd6 x1846780895197568/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234070.449740:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234070.449743:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234070.449745:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234070.449749:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6631 00000100:00080000:3.0:1761234070.449751:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234070.449755:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234070.449759:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761234070.449770:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234070.449771:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090702bc4 x1846780895197632/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234070.449782:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234070.449785:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234070.449787:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234070.449792:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6631 00000100:00080000:3.0:1761234070.449795:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234070.449799:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234070.449802:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234070.449813:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6fdd7f9 x1846780895197696/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761234070.449820:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234070/real 1761234070] req@000000003b40830d x1846780895197376/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234095 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234070.449826:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234070.449834:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234070.449837:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234070.449847:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6631 00000100:00080000:3.0:1761234070.449852:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234070.449856:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234070.449860:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234070.449873:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895197376/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234095 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234070.449875:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008e58c4b8 x1846780895197760/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234070.449912:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234070.449920:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234070.449929:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895197376:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761234070.449964:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895197440:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761234070.449994:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234070.450004:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234070/real 1761234070] req@00000000829a5595 x1846780895197440/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234095 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234070.450016:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895197440/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234095 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234070.450024:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234070.450026:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234070.450029:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895197440:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761234070.450034:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895197504:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761234070.450051:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234070.450057:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234070/real 1761234070] req@0000000089b8c4f1 x1846780895197504/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234095 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234070.450068:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895197504/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234095 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234070.450076:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234070.450077:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234070.450080:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895197504:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761234070.450084:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895197568:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761234070.450100:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234070.450106:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234070/real 1761234070] req@0000000070fa3bd6 x1846780895197568/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234095 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234070.450116:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895197568/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234095 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234070.450123:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234070.450124:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234070.450127:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895197568:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761234070.450131:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895197632:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:1.0:1761234070.450146:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234070.450152:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234070/real 1761234070] req@0000000090702bc4 x1846780895197632/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234095 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234070.450161:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895197632/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234095 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234070.450168:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234070.450172:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234070.450174:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895197632:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761234070.450179:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895197696:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761234070.450194:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234070.450231:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234070/real 1761234070] req@00000000b6fdd7f9 x1846780895197696/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234095 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234070.450241:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895197696/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234095 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234070.450258:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234070.450260:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234070.450262:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895197696:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761234070.450267:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895197760:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:1.0:1761234070.450288:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234070.450296:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234070/real 1761234070] req@000000008e58c4b8 x1846780895197760/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234095 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234070.450305:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895197760/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234095 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234070.450313:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234070.450315:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234070.450317:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895197760:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761234071.217434:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234072.241458:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234073.265470:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234074.289422:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234075.313609:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761234075.569424:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234075.569436:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761234075.569440:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234075.569453:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6636 00000100:00080000:1.0:1761234075.569457:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234075.569464:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761234075.569508:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df5b2270 x1846780895197824/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234075.569564:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234075.569568:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761234075.569570:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234075.569574:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6636 00000100:00080000:1.0:1761234075.569576:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234075.569579:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234075.569588:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234075.569601:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000df0b94a8 x1846780895197888/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234075.569605:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895197824:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761234075.569619:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234075.569623:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761234075.569624:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234075.569627:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6636 00000100:00080000:1.0:1761234075.569630:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234075.569633:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234075.569639:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234075.569650:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac65ecb7 x1846780895197952/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234075.569656:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234075.569659:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761234075.569660:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234075.569663:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6636 00000100:00080000:1.0:1761234075.569665:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234075.569667:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234075.569670:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234075.569679:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ffe86d04 x1846780895198016/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234075.569686:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234075.569688:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761234075.569689:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234075.569692:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6636 00000100:00080000:1.0:1761234075.569693:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234075.569696:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234075.569700:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234075.569710:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004361f727 x1846780895198080/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234075.569713:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895197888:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:1.0:1761234075.569734:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234075.569737:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761234075.569741:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234075.569748:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6636 00000100:00080000:1.0:1761234075.569751:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234075.569756:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234075.569759:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234075.569760:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac65ecb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895197952:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761234075.569778:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006a5e9865 x1846780895198144/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234075.569784:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234075.569787:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761234075.569791:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761234075.569791:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895198016:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:1.0:1761234075.569804:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6636 00000100:00080000:1.0:1761234075.569807:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234075.569813:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234075.569820:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234075.569836:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004361f727 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895198080:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761234075.569837:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d66930c2 x1846780895198208/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234075.569884:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006a5e9865 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895198144:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234075.569926:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895198208:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761234076.337471:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234077.361500:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234078.385429:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1761234078.641544:0:2207:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:0.0:1761234078.641583:0:2207:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1761234078.641597:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761234078.641629:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761234078.641636:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761234078.641645:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761234078.641654:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761234078.641663:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:0.0:1761234078.641674:0:2207:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:3.0:1761234078.641695:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234075/real 1761234078] req@00000000d66930c2 x1846780895198208/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234100 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234078.641723:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d66930c2 x1846780895198208/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234100 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234078.641735:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234078.641738:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234078.641742:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d66930c2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895198208:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234078.641749:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234075/real 1761234078] req@000000004361f727 x1846780895198080/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234100 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234078.641761:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004361f727 x1846780895198080/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234100 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234078.641769:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234078.641772:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234078.641775:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004361f727 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895198080:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234078.641781:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234075/real 1761234078] req@000000006a5e9865 x1846780895198144/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234100 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234078.641793:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006a5e9865 x1846780895198144/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234100 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234078.641801:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234078.641803:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234078.641806:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006a5e9865 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895198144:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234078.641813:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234075/real 1761234078] req@00000000df0b94a8 x1846780895197888/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234100 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234078.641825:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df0b94a8 x1846780895197888/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234100 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234078.641831:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234078.641834:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234078.641836:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df0b94a8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895197888:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234078.641841:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234075/real 1761234078] req@00000000ac65ecb7 x1846780895197952/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234100 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234078.641851:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac65ecb7 x1846780895197952/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234100 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234078.641857:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234078.641860:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234078.641863:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac65ecb7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895197952:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234078.641868:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234075/real 1761234078] req@00000000ffe86d04 x1846780895198016/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234100 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234078.641877:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ffe86d04 x1846780895198016/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234100 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234078.641884:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234078.641886:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234078.641889:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ffe86d04 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895198016:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234078.641897:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234075/real 1761234078] req@00000000df5b2270 x1846780895197824/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234100 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234078.641908:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000df5b2270 x1846780895197824/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234100 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234078.641924:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234078.641927:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234078.641931:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000df5b2270 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895197824:192.168.201.157@tcp:250:kworker/u8:3.0 00010000:00010000:3.0:1761234079.409416:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234080.433466:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761234080.689280:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234080.689291:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761234080.689295:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234080.689304:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6641 00000100:00080000:1.0:1761234080.689307:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234080.689313:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761234080.689342:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090ac513e x1846780895198272/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234080.689368:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234080.689372:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761234080.689373:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234080.689377:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6641 00000100:00080000:1.0:1761234080.689379:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234080.689382:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234080.689389:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234080.689398:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000053e52331 x1846780895198336/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761234080.689404:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895198272:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761234080.689407:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234080.689412:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761234080.689413:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234080.689417:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6641 00000100:00080000:1.0:1761234080.689419:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234080.689422:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234080.689426:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234080.689435:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000880f16b4 x1846780895198400/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234080.689441:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234080.689457:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761234080.689458:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234080.689462:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6641 00000100:00080000:1.0:1761234080.689465:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234080.689468:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234080.689473:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234080.689495:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cb5be845 x1846780895198464/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:2.0:1761234080.689504:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:1.0:1761234080.689505:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234080.689510:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761234080.689513:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234080.689518:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6641 00000100:00080000:1.0:1761234080.689521:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234080.689527:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:2.0:1761234080.689527:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234080/real 1761234080] req@0000000090ac513e x1846780895198272/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234105 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:1.0:1761234080.689536:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234080.689552:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a95774fb x1846780895198528/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761234080.689555:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090ac513e x1846780895198272/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234105 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234080.689573:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234080.689577:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234080.689579:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234080.689582:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:1.0:1761234080.689583:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1761234080.689590:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090ac513e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895198272:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761234080.689591:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6641 00000100:00080000:1.0:1761234080.689598:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234080.689604:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234080.689609:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234080.689621:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000000c736bb2 x1846780895198592/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761234080.689624:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895198336:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:1.0:1761234080.689638:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234080.689642:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761234080.689644:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234080.689649:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6641 00000100:00080000:1.0:1761234080.689652:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1761234080.689659:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:1.0:1761234080.689660:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234080.689667:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234080.689675:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234080/real 1761234080] req@0000000053e52331 x1846780895198336/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234105 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234080.689682:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000038209f5 x1846780895198656/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761234080.689696:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000053e52331 x1846780895198336/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234105 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234080.689706:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234080.689708:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234080.689714:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000053e52331 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895198336:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761234080.689720:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895198400:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761234080.689740:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234080.689748:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234080/real 1761234080] req@00000000880f16b4 x1846780895198400/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234105 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234080.689758:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000880f16b4 x1846780895198400/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234105 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234080.689765:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234080.689767:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234080.689769:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000880f16b4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895198400:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761234080.689775:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895198464:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761234080.689795:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234080.689803:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234080/real 1761234080] req@00000000cb5be845 x1846780895198464/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234105 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234080.689812:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cb5be845 x1846780895198464/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234105 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234080.689820:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234080.689822:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234080.689825:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cb5be845 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895198464:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761234080.689830:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895198528:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761234080.689851:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234080.689861:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234080/real 1761234080] req@00000000a95774fb x1846780895198528/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234105 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234080.689870:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a95774fb x1846780895198528/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234105 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234080.689878:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234080.689880:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234080.689883:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a95774fb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895198528:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761234080.689904:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895198592:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761234080.689926:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234080.689934:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234080/real 1761234080] req@000000000c736bb2 x1846780895198592/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234105 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234080.689945:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000000c736bb2 x1846780895198592/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234105 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234080.689952:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234080.689953:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234080.689956:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000000c736bb2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895198592:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761234080.689962:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895198656:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761234080.689979:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234080.689987:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234080/real 1761234080] req@00000000038209f5 x1846780895198656/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234105 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234080.689996:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000038209f5 x1846780895198656/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234105 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234080.690003:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234080.690005:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234080.690008:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000038209f5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895198656:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761234081.457350:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234082.481377:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234083.505410:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234084.529497:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234085.553504:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761234085.809333:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234085.809368:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761234085.809372:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234085.809385:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6646 00000100:00080000:1.0:1761234085.809405:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234085.809411:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761234085.809448:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000568f2a96 x1846780895198720/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234085.809477:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234085.809481:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761234085.809482:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234085.809502:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6646 00000100:00080000:1.0:1761234085.809504:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234085.809509:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234085.809518:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234085.809527:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895198720:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761234085.809532:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c652d0ad x1846780895198784/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234085.809547:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234085.809550:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761234085.809551:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234085.809555:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6646 00000100:00080000:1.0:1761234085.809557:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234085.809559:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234085.809564:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234085.809573:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007ba3340f x1846780895198848/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234085.809580:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234085.809583:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761234085.809584:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234085.809588:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6646 00000100:00080000:1.0:1761234085.809590:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234085.809592:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234085.809596:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1761234085.809602:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234085.809606:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d74b2e3c x1846780895198912/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234085.809614:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234085.809617:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761234085.809619:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234085.809622:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6646 00000100:00080000:1.0:1761234085.809624:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1761234085.809626:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234085/real 1761234085] req@00000000568f2a96 x1846780895198720/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234110 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234085.809631:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234085.809638:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234085.809648:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000568f2a96 x1846780895198720/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234110 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234085.809651:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001aedf9b1 x1846780895198976/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234085.809669:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234085.809670:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234085.809674:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:1.0:1761234085.809676:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761234085.809679:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1761234085.809681:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000568f2a96 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895198720:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761234085.809686:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6646 00000100:00080000:1.0:1761234085.809689:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234085.809693:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234085.809698:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234085.809708:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895198784:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761234085.809712:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a611ee34 x1846780895199040/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234085.809722:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234085.809725:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761234085.809726:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234085.809732:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6646 00000800:00000100:2.0:1761234085.809733:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:1.0:1761234085.809735:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234085.809739:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234085.809744:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234085.809745:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234085/real 1761234085] req@00000000c652d0ad x1846780895198784/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234110 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234085.809757:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000328368c8 x1846780895199104/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761234085.809759:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c652d0ad x1846780895198784/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234110 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234085.809780:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234085.809782:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234085.809784:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c652d0ad pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895198784:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761234085.809790:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895198848:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761234085.809803:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234085.809807:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234085/real 1761234085] req@000000007ba3340f x1846780895198848/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234110 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234085.809813:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007ba3340f x1846780895198848/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234110 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234085.809818:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234085.809819:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234085.809820:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007ba3340f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895198848:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761234085.809824:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895198912:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761234085.809835:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234085.809838:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234085/real 1761234085] req@00000000d74b2e3c x1846780895198912/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234110 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234085.809844:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d74b2e3c x1846780895198912/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234110 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234085.809849:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234085.809850:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234085.809851:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895198912:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761234085.809854:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895198976:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761234085.809863:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234085.809867:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234085/real 1761234085] req@000000001aedf9b1 x1846780895198976/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234110 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234085.809872:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001aedf9b1 x1846780895198976/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234110 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234085.809876:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234085.809878:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234085.809880:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895198976:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761234085.809891:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895199040:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761234085.809901:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234085.809904:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234085/real 1761234085] req@00000000a611ee34 x1846780895199040/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234110 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234085.809910:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a611ee34 x1846780895199040/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234110 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234085.809915:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234085.809916:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234085.809918:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895199040:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761234085.809921:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895199104:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761234085.809930:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234085.809933:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234085/real 1761234085] req@00000000328368c8 x1846780895199104/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234110 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234085.809939:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000328368c8 x1846780895199104/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234110 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234085.809943:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234085.809944:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234085.809945:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895199104:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761234086.577452:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234087.601427:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234088.625442:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234089.649442:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234090.673561:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761234090.929464:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234090.929473:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761234090.929477:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234090.929489:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6651 00000100:00080000:3.0:1761234090.929493:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234090.929500:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761234090.929537:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e99e5a12 x1846780895199168/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234090.929563:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234090.929567:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234090.929569:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234090.929573:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6651 00000100:00080000:3.0:1761234090.929577:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234090.929582:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234090.929592:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234090.929595:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895199168:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761234090.929607:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007adf99ab x1846780895199232/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234090.929619:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234090.929622:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234090.929623:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234090.929627:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6651 00000100:00080000:3.0:1761234090.929630:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234090.929633:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234090.929638:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234090.929647:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000080131ea7 x1846780895199296/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234090.929657:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1761234090.929661:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761234090.929665:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234090.929667:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234090.929673:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6651 00000100:00080000:3.0:1761234090.929676:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234090.929679:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:2.0:1761234090.929683:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234090/real 1761234090] req@00000000e99e5a12 x1846780895199168/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234115 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:3.0:1761234090.929685:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234090.929700:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895199168/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234115 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234090.929704:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008e58c4b8 x1846780895199360/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234090.929714:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234090.929718:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:3.0:1761234090.929719:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1761234090.929723:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895199168:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:3.0:1761234090.929728:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234090.929731:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234090.929736:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6651 00000100:00080000:3.0:1761234090.929739:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234090.929743:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234090.929748:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234090.929750:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895199232:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234090.929765:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6fdd7f9 x1846780895199424/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:2.0:1761234090.929772:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234090.929779:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234090/real 1761234090] req@000000007adf99ab x1846780895199232/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234115 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234090.929782:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234090.929791:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234090.929794:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1761234090.929796:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895199232/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234115 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234090.929802:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6651 00000100:00080000:2.0:1761234090.929805:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234090.929807:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234090.929808:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234090.929813:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895199232:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:3.0:1761234090.929816:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:2.0:1761234090.929821:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895199296:192.168.201.157@tcp:8:kworker/u8:3.0 00000008:00100000:3.0:1761234090.929825:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1761234090.929838:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234090.929843:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090702bc4 x1846780895199488/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761234090.929846:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234090/real 1761234090] req@0000000080131ea7 x1846780895199296/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234115 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234090.929874:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895199296/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234115 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234090.929875:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234090.929884:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234090.929886:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234090.929888:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234090.929889:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234090.929893:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895199296:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1761234090.929898:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6651 00000100:00100000:2.0:1761234090.929900:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895199360:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:3.0:1761234090.929903:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234090.929910:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:2.0:1761234090.929914:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:3.0:1761234090.929917:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234090.929921:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234090/real 1761234090] req@000000008e58c4b8 x1846780895199360/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234115 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234090.929929:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895199360/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234115 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234090.929935:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000070fa3bd6 x1846780895199552/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234090.929937:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234090.929939:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234090.929945:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895199360:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761234090.929963:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895199424:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761234090.929976:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234090.929982:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234090/real 1761234090] req@00000000b6fdd7f9 x1846780895199424/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234115 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234090.929988:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895199424/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234115 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234090.929993:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234090.929995:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234090.929996:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895199424:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761234090.930000:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895199488:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761234090.930008:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234090.930012:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234090/real 1761234090] req@0000000090702bc4 x1846780895199488/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234115 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234090.930017:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895199488/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234115 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234090.930022:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234090.930024:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234090.930025:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895199488:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761234090.930029:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895199552:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761234090.930037:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234090.930041:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234090/real 1761234090] req@0000000070fa3bd6 x1846780895199552/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234115 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234090.930046:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895199552/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234115 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234090.930050:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234090.930052:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234090.930053:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895199552:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761234091.697339:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234092.721441:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234093.745369:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234094.769429:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234095.793367:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761234096.049305:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234096.049315:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761234096.049319:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234096.049330:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6656 00000100:00080000:2.0:1761234096.049334:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234096.049341:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761234096.049377:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895199616/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234096.049403:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234096.049407:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234096.049408:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234096.049412:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6656 00000100:00080000:2.0:1761234096.049416:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234096.049420:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234096.049429:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234096.049449:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895199616:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761234096.049454:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895199680/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234096.049466:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234096.049470:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234096.049471:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234096.049475:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6656 00000100:00080000:2.0:1761234096.049478:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234096.049481:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234096.049486:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234096.049496:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895199744/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234096.049515:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234096.049518:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234096.049520:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234096.049523:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6656 00000100:00080000:2.0:1761234096.049525:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234096.049531:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:3.0:1761234096.049531:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:2.0:1761234096.049538:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234096.049549:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895199808/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234096.049556:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234096/real 1761234096] req@000000004ae5bc9b x1846780895199616/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234121 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234096.049563:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234096.049573:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234096.049575:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234096.049580:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6656 00000100:00100000:3.0:1761234096.049582:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895199616/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234121 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234096.049584:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234096.049592:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:3.0:1761234096.049595:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1761234096.049599:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761234096.049600:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234096.049605:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895199616:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761234096.049615:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895199872/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234096.049627:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761234096.049629:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895199680:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:2.0:1761234096.049631:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234096.049634:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234096.049639:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6656 00000100:00080000:2.0:1761234096.049640:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234096.049644:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234096.049648:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761234096.049653:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234096.049660:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895199936/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234096.049664:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234096/real 1761234096] req@00000000ac7d8d5b x1846780895199680/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234121 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234096.049679:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234096.049684:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234096.049685:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761234096.049688:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895199680/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234121 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234096.049691:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6656 00000100:00080000:2.0:1761234096.049694:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234096.049698:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234096.049703:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1761234096.049706:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234096.049711:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234096.049713:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895200000/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234096.049718:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895199680:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761234096.049736:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895199744:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761234096.049751:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234096.049755:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234096/real 1761234096] req@00000000f4e415d3 x1846780895199744/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234121 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234096.049762:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895199744/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234121 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234096.049767:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234096.049768:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234096.049770:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895199744:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761234096.049773:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895199808:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761234096.049783:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234096.049787:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234096/real 1761234096] req@0000000099b7e5d9 x1846780895199808/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234121 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234096.049794:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895199808/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234121 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234096.049799:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234096.049800:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234096.049801:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895199808:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761234096.049824:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895199872:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761234096.049834:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234096.049839:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234096/real 1761234096] req@00000000c10cebe6 x1846780895199872/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234121 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234096.049846:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895199872/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234121 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234096.049851:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234096.049852:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234096.049854:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895199872:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761234096.049857:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895199936:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761234096.049866:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234096.049871:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234096/real 1761234096] req@00000000a1e8c6bb x1846780895199936/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234121 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234096.049877:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895199936/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234121 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234096.049882:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234096.049883:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234096.049884:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895199936:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761234096.049888:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895200000:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761234096.049897:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234096.049900:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234096/real 1761234096] req@000000008ebccab5 x1846780895200000/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234121 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234096.049906:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895200000/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234121 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234096.049910:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234096.049912:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234096.049914:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895200000:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761234096.817442:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234097.841403:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234098.865355:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234099.889397:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234100.913422:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761234101.169323:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234101.169337:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761234101.169342:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234101.169354:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6662 00000100:00080000:2.0:1761234101.169359:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234101.169386:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761234101.169426:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895200064/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234101.169455:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234101.169459:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234101.169460:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234101.169464:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6662 00000100:00080000:2.0:1761234101.169466:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234101.169469:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234101.169478:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234101.169487:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895200128/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234101.169494:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234101.169496:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234101.169497:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234101.169500:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6662 00000100:00080000:2.0:1761234101.169502:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234101.169504:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234101.169509:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234101.169515:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895200192/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234101.169522:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234101.169524:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234101.169526:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234101.169529:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6662 00000100:00080000:2.0:1761234101.169531:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234101.169535:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234101.169538:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234101.169544:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895200256/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234101.169551:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234101.169554:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234101.169555:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761234101.169561:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895200064:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761234101.169562:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6662 00000100:00080000:2.0:1761234101.169565:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234101.169571:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234101.169575:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234101.169585:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895200320/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234101.169592:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234101.169595:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234101.169597:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234101.169601:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6662 00000100:00080000:2.0:1761234101.169603:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234101.169606:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234101.169610:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234101.169618:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895200384/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234101.169624:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234101.169627:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234101.169628:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234101.169631:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6662 00000100:00080000:2.0:1761234101.169633:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234101.169635:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234101.169640:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761234101.169642:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234101.169652:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895200448/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234101.169670:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234101/real 1761234101] req@000000008ebccab5 x1846780895200064/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234126 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234101.169688:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895200064/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234126 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234101.169697:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234101.169699:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234101.169703:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895200064:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761234101.169710:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895200128:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761234101.169729:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234101.169735:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234101/real 1761234101] req@00000000a1e8c6bb x1846780895200128/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234126 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234101.169744:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895200128/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234126 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234101.169750:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234101.169752:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234101.169755:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895200128:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761234101.169760:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895200192:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761234101.169774:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234101.169779:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234101/real 1761234101] req@00000000c10cebe6 x1846780895200192/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234126 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234101.169788:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895200192/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234126 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234101.169794:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234101.169795:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234101.169797:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895200192:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761234101.169825:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895200256:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761234101.169838:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234101.169843:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234101/real 1761234101] req@0000000099b7e5d9 x1846780895200256/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234126 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234101.169851:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895200256/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234126 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234101.169857:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234101.169858:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234101.169861:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895200256:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761234101.169865:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895200320:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761234101.169877:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234101.169883:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234101/real 1761234101] req@00000000f4e415d3 x1846780895200320/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234126 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234101.169892:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895200320/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234126 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234101.169898:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234101.169899:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234101.169901:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895200320:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761234101.169906:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895200384:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761234101.169918:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234101.169923:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234101/real 1761234101] req@00000000ac7d8d5b x1846780895200384/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234126 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234101.169930:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895200384/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234126 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234101.169936:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234101.169938:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234101.169940:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895200384:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761234101.169944:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895200448:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761234101.169964:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234101.169974:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234101/real 1761234101] req@000000004ae5bc9b x1846780895200448/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234126 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234101.169987:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895200448/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234126 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234101.169997:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234101.170000:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234101.170004:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895200448:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761234101.937388:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234102.961455:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234103.985436:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234105.009401:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234106.033425:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761234106.289350:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234106.289360:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761234106.289364:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234106.289373:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6667 00000100:00080000:3.0:1761234106.289376:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234106.289382:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761234106.289412:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000089b8c4f1 x1846780895200512/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234106.289432:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234106.289435:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234106.289436:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234106.289438:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6667 00000100:00080000:3.0:1761234106.289440:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234106.289442:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234106.289449:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234106.289461:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000829a5595 x1846780895200576/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761234106.289467:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895200512:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:3.0:1761234106.289479:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234106.289483:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234106.289484:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234106.289490:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6667 00000100:00080000:3.0:1761234106.289493:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234106.289496:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234106.289502:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234106.289515:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b40830d x1846780895200640/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234106.289524:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1761234106.289531:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761234106.289543:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234106.289545:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234106.289549:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6667 00000100:00080000:3.0:1761234106.289552:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1761234106.289558:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234106/real 1761234106] req@0000000089b8c4f1 x1846780895200512/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234131 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234106.289559:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234106.289565:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234106.289580:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002625ec0b x1846780895200704/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761234106.289586:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895200512/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234131 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234106.289602:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234106.289606:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234106.289609:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234106.289611:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:3.0:1761234106.289613:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1761234106.289618:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895200512:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:3.0:1761234106.289621:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6667 00000100:00080000:3.0:1761234106.289626:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234106.289631:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234106.289636:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234106.289648:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895200576:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761234106.289649:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c9993cd4 x1846780895200768/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234106.289664:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234106.289667:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234106.289669:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234106.289673:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6667 00000100:00080000:3.0:1761234106.289675:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1761234106.289680:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761234106.289681:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234106.289688:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234106.289694:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234106/real 1761234106] req@00000000829a5595 x1846780895200576/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234131 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234106.289701:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a0ba65f x1846780895200832/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761234106.289712:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895200576/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234131 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234106.289713:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234106.289721:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234106.289722:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234106.289729:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234106.289730:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6667 00000100:00080000:2.0:1761234106.289735:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00080000:3.0:1761234106.289736:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234106.289743:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:2.0:1761234106.289746:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895200576:192.168.201.157@tcp:38:kworker/u8:2.0 00000008:00100000:3.0:1761234106.289750:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234106.289755:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895200640:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761234106.289765:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e55ffa x1846780895200896/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:2.0:1761234106.289776:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234106.289786:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234106/real 1761234106] req@000000003b40830d x1846780895200640/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234131 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234106.289794:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895200640/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234131 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234106.289799:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234106.289800:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234106.289802:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895200640:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761234106.289806:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895200704:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761234106.289816:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234106.289819:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234106/real 1761234106] req@000000002625ec0b x1846780895200704/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234131 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234106.289826:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895200704/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234131 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234106.289830:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234106.289831:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234106.289833:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895200704:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761234106.289846:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895200768:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761234106.289857:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234106.289861:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234106/real 1761234106] req@00000000c9993cd4 x1846780895200768/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234131 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234106.289867:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895200768/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234131 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234106.289871:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234106.289873:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234106.289874:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895200768:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761234106.289877:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895200832:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761234106.289886:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234106.289889:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234106/real 1761234106] req@000000002a0ba65f x1846780895200832/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234131 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234106.289895:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895200832/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234131 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234106.289898:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234106.289900:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234106.289902:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895200832:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761234106.289905:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895200896:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761234106.289913:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234106.289917:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234106/real 1761234106] req@0000000057e55ffa x1846780895200896/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234131 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234106.289922:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895200896/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234131 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234106.289926:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234106.289927:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234106.289929:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895200896:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761234107.057461:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234108.081430:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234109.105379:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234110.129436:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234111.153404:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761234111.409294:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234111.409306:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761234111.409309:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234111.409321:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6672 00000100:00080000:3.0:1761234111.409325:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234111.409333:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761234111.409366:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007aee79ba x1846780895200960/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234111.409393:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234111.409395:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234111.409397:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234111.409399:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6672 00000100:00080000:3.0:1761234111.409401:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234111.409404:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234111.409411:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234111.409418:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009706a782 x1846780895201024/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234111.409424:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234111.409426:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234111.409427:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234111.409429:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6672 00000100:00080000:3.0:1761234111.409430:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234111.409432:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234111.409435:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234111.409440:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000bbf6af7b x1846780895201088/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234111.409445:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234111.409447:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234111.409448:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234111.409450:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6672 00000100:00080000:3.0:1761234111.409453:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234111.409455:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234111.409458:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234111.409463:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895200960:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761234111.409467:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e55ffa x1846780895201152/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234111.409476:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234111.409478:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234111.409479:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234111.409481:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6672 00000100:00080000:3.0:1761234111.409483:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234111.409485:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234111.409488:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234111.409503:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a0ba65f x1846780895201216/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234111.409509:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234111.409511:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234111.409511:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234111.409514:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6672 00000100:00080000:3.0:1761234111.409515:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234111.409518:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234111.409521:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234111.409526:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c9993cd4 x1846780895201280/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234111.409531:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234111.409534:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234111.409535:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234111.409537:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6672 00000100:00080000:3.0:1761234111.409538:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1761234111.409543:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761234111.409543:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234111.409546:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234111.409555:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002625ec0b x1846780895201344/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761234111.409566:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234111/real 1761234111] req@000000007aee79ba x1846780895200960/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234136 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234111.409581:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895200960/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234136 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234111.409588:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234111.409591:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234111.409594:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895200960:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761234111.409598:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895201024:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761234111.409614:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234111.409619:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234111/real 1761234111] req@000000009706a782 x1846780895201024/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234136 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234111.409626:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895201024/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234136 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234111.409630:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234111.409632:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234111.409633:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895201024:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761234111.409637:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895201088:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761234111.409648:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234111.409652:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234111/real 1761234111] req@00000000bbf6af7b x1846780895201088/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234136 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234111.409662:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895201088/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234136 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234111.409669:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234111.409671:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234111.409674:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895201088:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761234111.409702:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895201152:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761234111.409719:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234111.409725:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234111/real 1761234111] req@0000000057e55ffa x1846780895201152/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234136 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234111.409731:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895201152/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234136 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234111.409737:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234111.409738:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234111.409740:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895201152:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761234111.409743:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895201216:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761234111.409753:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234111.409757:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234111/real 1761234111] req@000000002a0ba65f x1846780895201216/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234136 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234111.409763:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895201216/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234136 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234111.409768:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234111.409769:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234111.409771:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895201216:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761234111.409774:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895201280:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761234111.409782:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234111.409786:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234111/real 1761234111] req@00000000c9993cd4 x1846780895201280/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234136 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234111.409792:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895201280/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234136 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234111.409797:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234111.409798:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234111.409799:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895201280:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761234111.409803:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895201344:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761234111.409812:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234111.409816:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234111/real 1761234111] req@000000002625ec0b x1846780895201344/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234136 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234111.409822:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895201344/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234136 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234111.409826:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234111.409827:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234111.409829:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895201344:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761234112.177471:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234113.201388:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234114.225476:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234115.249445:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234116.273489:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761234116.529369:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234116.529382:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761234116.529386:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234116.529398:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6677 00000100:00080000:3.0:1761234116.529403:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234116.529413:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761234116.529452:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b40830d x1846780895201408/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234116.529478:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234116.529480:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234116.529482:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234116.529484:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6677 00000100:00080000:3.0:1761234116.529486:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234116.529489:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234116.529497:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234116.529506:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000829a5595 x1846780895201472/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234116.529512:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234116.529514:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234116.529515:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234116.529517:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6677 00000100:00080000:3.0:1761234116.529518:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234116.529520:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234116.529523:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234116.529528:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000089b8c4f1 x1846780895201536/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234116.529534:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234116.529536:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234116.529537:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234116.529544:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6677 00000100:00080000:3.0:1761234116.529547:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234116.529551:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234116.529556:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234116.529567:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000070fa3bd6 x1846780895201600/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761234116.529576:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895201408:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:3.0:1761234116.529589:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234116.529595:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234116.529596:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234116.529604:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6677 00000100:00080000:3.0:1761234116.529607:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234116.529611:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234116.529618:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234116.529636:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090702bc4 x1846780895201664/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234116.529646:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234116.529650:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234116.529652:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234116.529656:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6677 00000100:00080000:3.0:1761234116.529659:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234116.529664:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:2.0:1761234116.529671:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:3.0:1761234116.529676:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234116.529695:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6fdd7f9 x1846780895201728/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761234116.529702:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234116/real 1761234116] req@000000003b40830d x1846780895201408/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234141 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234116.529737:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234116.529742:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1761234116.529746:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895201408/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234141 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234116.529746:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234116.529777:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6677 00000100:00080000:3.0:1761234116.529781:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234116.529786:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:2.0:1761234116.529797:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1761234116.529798:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1761234116.529803:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234116.529809:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895201408:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761234116.529817:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895201472:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234116.529822:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008e58c4b8 x1846780895201792/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:2.0:1761234116.529843:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234116.529857:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234116/real 1761234116] req@00000000829a5595 x1846780895201472/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234141 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234116.529873:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895201472/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234141 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234116.529878:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234116.529880:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234116.529882:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895201472:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761234116.529886:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895201536:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761234116.529897:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234116.529901:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234116/real 1761234116] req@0000000089b8c4f1 x1846780895201536/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234141 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234116.529907:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895201536/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234141 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234116.529912:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234116.529913:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234116.529915:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895201536:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761234116.529941:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895201600:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761234116.529951:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234116.529955:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234116/real 1761234116] req@0000000070fa3bd6 x1846780895201600/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234141 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234116.529960:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895201600/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234141 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234116.529965:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234116.529967:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234116.529969:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895201600:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761234116.529973:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895201664:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761234116.529983:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234116.529987:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234116/real 1761234116] req@0000000090702bc4 x1846780895201664/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234141 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234116.529992:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895201664/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234141 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234116.529997:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234116.529999:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234116.530001:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895201664:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761234116.530004:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895201728:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761234116.530013:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234116.530017:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234116/real 1761234116] req@00000000b6fdd7f9 x1846780895201728/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234141 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234116.530022:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895201728/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234141 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234116.530026:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234116.530028:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234116.530030:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895201728:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761234116.530033:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895201792:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761234116.530045:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234116.530051:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234116/real 1761234116] req@000000008e58c4b8 x1846780895201792/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234141 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234116.530062:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895201792/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234141 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234116.530070:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234116.530072:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234116.530075:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895201792:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761234117.297521:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234118.321475:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234119.345493:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234120.369389:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234121.393452:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761234121.649346:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234121.649359:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761234121.649363:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234121.649374:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6682 00000100:00080000:3.0:1761234121.649377:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234121.649384:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761234121.649418:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000080131ea7 x1846780895201856/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234121.649442:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234121.649445:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234121.649446:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234121.649448:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6682 00000100:00080000:3.0:1761234121.649450:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234121.649453:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234121.649460:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234121.649467:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007adf99ab x1846780895201920/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234121.649473:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234121.649475:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234121.649477:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234121.649480:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6682 00000100:00080000:3.0:1761234121.649484:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234121.649487:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:2.0:1761234121.649492:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895201856:192.168.201.157@tcp:250:kworker/u8:3.0 00000008:00100000:3.0:1761234121.649494:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234121.649509:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e99e5a12 x1846780895201984/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234121.649520:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234121.649524:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234121.649526:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234121.649530:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6682 00000100:00080000:3.0:1761234121.649532:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234121.649536:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234121.649541:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234121.649553:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008e58c4b8 x1846780895202048/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:2.0:1761234121.649561:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761234121.649567:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234121.649570:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234121.649572:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234121.649576:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6682 00000100:00080000:3.0:1761234121.649577:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234121.649580:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234121.649583:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234121.649586:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234121/real 1761234121] req@0000000080131ea7 x1846780895201856/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234146 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234121.649593:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6fdd7f9 x1846780895202112/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234121.649602:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234121.649605:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1761234121.649607:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895201856/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234146 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234121.649607:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234121.649615:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6682 00000100:00080000:3.0:1761234121.649619:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234121.649623:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:2.0:1761234121.649625:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1761234121.649626:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1761234121.649628:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234121.649633:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895201856:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761234121.649637:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090702bc4 x1846780895202176/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761234121.649643:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895201920:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:3.0:1761234121.649645:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234121.649651:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234121.649652:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234121.649657:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6682 00000100:00080000:3.0:1761234121.649658:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234121.649661:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234121.649664:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234121.649688:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000070fa3bd6 x1846780895202240/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:2.0:1761234121.649695:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234121.649708:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234121/real 1761234121] req@000000007adf99ab x1846780895201920/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234146 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234121.649716:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895201920/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234146 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234121.649722:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234121.649724:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234121.649727:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895201920:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761234121.649746:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895201984:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761234121.649760:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234121.649765:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234121/real 1761234121] req@00000000e99e5a12 x1846780895201984/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234146 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234121.649772:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895201984/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234146 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234121.649777:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234121.649778:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234121.649780:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895201984:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761234121.649788:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895202048:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761234121.649799:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234121.649803:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234121/real 1761234121] req@000000008e58c4b8 x1846780895202048/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234146 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234121.649810:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895202048/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234146 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234121.649815:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234121.649816:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234121.649818:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895202048:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761234121.649822:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895202112:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:2.0:1761234121.649832:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234121.649836:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234121/real 1761234121] req@00000000b6fdd7f9 x1846780895202112/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234146 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234121.649843:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895202112/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234146 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234121.649847:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234121.649849:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234121.649850:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895202112:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761234121.649853:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895202176:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761234121.649864:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234121.649868:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234121/real 1761234121] req@0000000090702bc4 x1846780895202176/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234146 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234121.649874:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895202176/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234146 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234121.649879:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234121.649880:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234121.649881:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895202176:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761234121.649885:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895202240:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:2.0:1761234121.649895:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234121.649898:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234121/real 1761234121] req@0000000070fa3bd6 x1846780895202240/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234146 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234121.649905:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895202240/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234146 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234121.649910:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234121.649911:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234121.649913:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895202240:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761234122.417345:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234123.441426:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234124.465411:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234125.489365:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234126.513400:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761234126.769369:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234126.769380:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761234126.769384:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234126.769410:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6687 00000100:00080000:3.0:1761234126.769413:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234126.769419:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761234126.769454:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000089b8c4f1 x1846780895202304/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234126.769479:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234126.769482:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234126.769483:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234126.769485:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6687 00000100:00080000:3.0:1761234126.769487:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234126.769490:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234126.769498:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234126.769508:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000829a5595 x1846780895202368/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761234126.769518:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895202304:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:3.0:1761234126.769531:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234126.769535:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234126.769537:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234126.769541:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6687 00000100:00080000:3.0:1761234126.769544:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234126.769547:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234126.769553:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234126.769565:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b40830d x1846780895202432/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234126.769588:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234126.769591:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234126.769592:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234126.769596:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6687 00000100:00080000:3.0:1761234126.769600:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1761234126.769601:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761234126.769606:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234126.769623:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234126.769635:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002625ec0b x1846780895202496/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761234126.769641:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234126/real 1761234126] req@0000000089b8c4f1 x1846780895202304/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234151 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234126.769647:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234126.769656:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234126.769658:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234126.769664:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6687 00000100:00080000:3.0:1761234126.769667:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1761234126.769670:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895202304/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234151 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234126.769673:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234126.769681:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1761234126.769683:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234126.769687:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234126.769691:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895202304:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761234126.769695:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c9993cd4 x1846780895202560/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234126.769706:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234126.769708:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234126.769709:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234126.769713:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6687 00000100:00080000:3.0:1761234126.769715:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1761234126.769717:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895202368:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:3.0:1761234126.769721:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234126.769727:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234126.769737:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a0ba65f x1846780895202624/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:2.0:1761234126.769744:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761234126.769745:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234126.769749:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234126.769750:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1761234126.769753:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234126/real 1761234126] req@00000000829a5595 x1846780895202368/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234151 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234126.769755:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6687 00000100:00080000:3.0:1761234126.769758:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234126.769762:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234126.769767:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234126.769772:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895202368/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234151 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234126.769778:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e55ffa x1846780895202688/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234126.769782:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234126.769786:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234126.769790:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895202368:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761234126.769795:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895202432:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761234126.769809:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234126.769813:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234126/real 1761234126] req@000000003b40830d x1846780895202432/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234151 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234126.769820:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895202432/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234151 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234126.769824:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234126.769825:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234126.769826:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895202432:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761234126.769830:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895202496:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761234126.769839:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234126.769843:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234126/real 1761234126] req@000000002625ec0b x1846780895202496/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234151 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234126.769849:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895202496/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234151 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234126.769853:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234126.769854:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234126.769856:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895202496:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761234126.769859:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895202560:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761234126.769871:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234126.769875:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234126/real 1761234126] req@00000000c9993cd4 x1846780895202560/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234151 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234126.769881:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895202560/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234151 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234126.769886:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234126.769887:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234126.769888:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895202560:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761234126.769900:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895202624:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761234126.769910:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234126.769914:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234126/real 1761234126] req@000000002a0ba65f x1846780895202624/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234151 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234126.769919:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895202624/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234151 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234126.769923:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234126.769925:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234126.769926:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895202624:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761234126.769930:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895202688:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761234126.769938:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234126.769942:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234126/real 1761234126] req@0000000057e55ffa x1846780895202688/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234151 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234126.769948:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895202688/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234151 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234126.769953:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234126.769954:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234126.769956:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895202688:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761234127.537500:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234128.561376:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234129.585458:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234130.609371:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234131.633524:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761234131.889361:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234131.889370:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761234131.889373:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234131.889383:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6692 00000100:00080000:2.0:1761234131.889385:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234131.889391:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761234131.889422:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895202752/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234131.889445:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234131.889449:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234131.889450:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234131.889454:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6692 00000100:00080000:2.0:1761234131.889456:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234131.889459:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234131.889469:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234131.889495:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895202816/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234131.889514:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234131.889517:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234131.889519:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234131.889523:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6692 00000100:00100000:3.0:1761234131.889526:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895202752:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761234131.889532:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234131.889535:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234131.889540:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234131.889552:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895202880/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234131.889560:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234131.889563:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234131.889564:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234131.889567:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6692 00000100:00080000:2.0:1761234131.889569:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234131.889572:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234131.889575:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234131.889582:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895202944/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234131.889589:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1761234131.889590:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761234131.889593:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234131.889594:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234131.889596:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6692 00000100:00080000:2.0:1761234131.889598:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234131.889601:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234131.889605:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234131.889611:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895203008/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234131.889612:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234131/real 1761234131] req@00000000fd4aa3a2 x1846780895202752/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234156 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234131.889625:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234131.889627:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1761234131.889627:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895202752/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234156 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234131.889629:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234131.889632:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6692 00000100:00080000:2.0:1761234131.889634:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234131.889637:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:3.0:1761234131.889637:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234131.889640:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000008:00100000:2.0:1761234131.889641:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234131.889644:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895202752:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761234131.889647:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895203072/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234131.889650:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895202816:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00080000:2.0:1761234131.889655:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234131.889658:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234131.889659:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234131.889663:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6692 00000100:00080000:2.0:1761234131.889664:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234131.889667:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:3.0:1761234131.889669:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:2.0:1761234131.889671:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234131.889677:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234131/real 1761234131] req@000000006c1041ba x1846780895202816/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234156 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234131.889679:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895203136/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234131.889688:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895202816/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234156 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234131.889695:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234131.889698:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234131.889701:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895202816:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761234131.889726:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895202880:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761234131.889745:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234131.889750:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234131/real 1761234131] req@000000006d8c3eb0 x1846780895202880/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234156 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234131.889768:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895202880/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234156 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234131.889773:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234131.889774:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234131.889776:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895202880:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761234131.889779:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895202944:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761234131.889789:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234131.889792:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234131/real 1761234131] req@0000000088939481 x1846780895202944/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234156 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234131.889799:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895202944/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234156 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234131.889803:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234131.889804:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234131.889805:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895202944:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761234131.889809:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895203008:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761234131.889818:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234131.889822:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234131/real 1761234131] req@000000006ff1fdbc x1846780895203008/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234156 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234131.889828:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895203008/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234156 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234131.889833:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234131.889834:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234131.889835:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895203008:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761234131.889838:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895203072:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761234131.889847:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234131.889850:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234131/real 1761234131] req@00000000135e9a6e x1846780895203072/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234156 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234131.889856:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895203072/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234156 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234131.889860:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234131.889861:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234131.889863:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895203072:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761234131.889866:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895203136:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761234131.889875:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234131.889878:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234131/real 1761234131] req@00000000fd3935a6 x1846780895203136/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234156 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234131.889884:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895203136/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234156 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234131.889889:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234131.889890:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234131.889891:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895203136:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761234132.657411:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234133.681479:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234134.705483:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234135.729444:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234136.753456:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761234137.009299:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234137.009309:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761234137.009313:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234137.009324:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6697 00000100:00080000:2.0:1761234137.009328:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234137.009336:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761234137.009387:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895203200/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234137.009409:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234137.009412:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234137.009413:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234137.009416:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6697 00000100:00080000:2.0:1761234137.009417:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234137.009420:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234137.009428:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234137.009436:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895203264/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234137.009442:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234137.009445:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234137.009446:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234137.009448:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6697 00000100:00080000:2.0:1761234137.009449:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234137.009451:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234137.009453:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234137.009458:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895203328/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234137.009464:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234137.009466:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234137.009467:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234137.009470:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6697 00000100:00080000:2.0:1761234137.009472:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234137.009475:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234137.009478:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234137.009485:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895203392/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234137.009491:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895203200:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761234137.009493:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234137.009500:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234137.009502:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234137.009506:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6697 00000100:00080000:2.0:1761234137.009507:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234137.009510:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234137.009513:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234137.009522:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895203456/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234137.009527:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234137.009530:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234137.009531:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234137.009533:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6697 00000100:00080000:2.0:1761234137.009534:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234137.009537:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234137.009539:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234137.009546:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895203520/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234137.009555:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234137.009557:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234137.009558:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234137.009560:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6697 00000100:00080000:2.0:1761234137.009561:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234137.009566:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:3.0:1761234137.009568:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:2.0:1761234137.009570:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234137.009580:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895203584/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234137.009592:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234137/real 1761234137] req@000000007dcf84ae x1846780895203200/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234162 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234137.009607:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895203200/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234162 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234137.009615:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234137.009617:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234137.009620:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895203200:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761234137.009625:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895203264:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761234137.009641:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234137.009646:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234137/real 1761234137] req@000000004f29ca26 x1846780895203264/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234162 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234137.009653:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895203264/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234162 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234137.009662:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234137.009664:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234137.009666:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895203264:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234137.009669:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895203328:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234137.009680:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234137.009684:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234137/real 1761234137] req@000000007091f547 x1846780895203328/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234162 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234137.009690:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895203328/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234162 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234137.009695:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234137.009696:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234137.009698:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895203328:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234137.009721:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895203392:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234137.009732:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234137.009736:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234137/real 1761234137] req@0000000064a2cea0 x1846780895203392/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234162 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234137.009742:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895203392/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234162 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234137.009747:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234137.009748:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234137.009750:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895203392:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234137.009754:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895203456:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761234137.009764:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234137.009768:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234137/real 1761234137] req@0000000033765045 x1846780895203456/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234162 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234137.009775:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895203456/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234162 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234137.009780:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234137.009781:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234137.009783:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895203456:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234137.009786:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895203520:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234137.009796:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234137.009800:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234137/real 1761234137] req@000000009d315fa5 x1846780895203520/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234162 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234137.009806:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895203520/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234162 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234137.009811:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234137.009812:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234137.009814:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895203520:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234137.009817:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895203584:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234137.009827:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234137.009830:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234137/real 1761234137] req@0000000040d5f361 x1846780895203584/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234162 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234137.009837:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895203584/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234162 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234137.009842:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234137.009843:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234137.009845:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895203584:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761234137.777477:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234138.801428:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234139.825489:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234140.849407:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234141.873407:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761234142.129364:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234142.129374:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761234142.129377:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234142.129387:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6703 00000100:00080000:2.0:1761234142.129389:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234142.129395:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761234142.129428:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895203648/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234142.129453:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234142.129456:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234142.129458:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234142.129461:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6703 00000100:00080000:2.0:1761234142.129463:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234142.129466:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234142.129478:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234142.129489:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895203712/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234142.129498:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234142.129501:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234142.129502:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234142.129506:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6703 00000100:00080000:2.0:1761234142.129508:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234142.129511:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234142.129515:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234142.129522:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895203776/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234142.129522:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895203648:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761234142.129530:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234142.129533:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234142.129534:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234142.129538:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6703 00000100:00080000:2.0:1761234142.129539:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234142.129542:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234142.129545:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234142.129553:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895203840/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234142.129559:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234142.129562:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234142.129563:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234142.129567:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6703 00000100:00080000:2.0:1761234142.129568:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234142.129571:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234142.129574:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234142.129581:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895203904/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234142.129588:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234142.129591:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234142.129592:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234142.129596:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6703 00000100:00080000:2.0:1761234142.129597:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234142.129600:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234142.129604:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234142.129605:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895203712:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761234142.129612:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895203968/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234142.129619:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234142.129622:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234142.129624:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234142.129629:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6703 00000100:00080000:2.0:1761234142.129631:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234142.129635:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234142.129639:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234142.129647:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895203776:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761234142.129653:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895204032/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234142.129675:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895203840:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234142.129698:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895203904:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234142.129726:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895203968:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234142.129761:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895204032:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761234142.897489:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234143.921451:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234144.945448:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1761234145.201665:0:2208:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:2.0:1761234145.201717:0:2208:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1761234145.201733:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761234145.201777:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761234145.201785:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761234145.201793:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761234145.201802:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761234145.201812:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761234145.201822:0:2208:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:3.0:1761234145.201839:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234142/real 1761234145] req@000000007dcf84ae x1846780895204032/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234167 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234145.201868:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895204032/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234167 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234145.201881:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234145.201886:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234145.201891:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895204032:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234145.201898:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234142/real 1761234145] req@0000000033765045 x1846780895203776/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234167 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234145.201908:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895203776/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234167 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234145.201914:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234145.201917:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234145.201927:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895203776:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234145.201933:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234142/real 1761234145] req@0000000064a2cea0 x1846780895203840/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234167 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234145.201947:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895203840/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234167 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234145.201955:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234145.201957:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234145.201960:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895203840:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234145.201966:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234142/real 1761234145] req@000000007091f547 x1846780895203904/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234167 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234145.201976:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895203904/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234167 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234145.201983:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234145.201985:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234145.201989:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895203904:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234145.201994:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234142/real 1761234145] req@000000004f29ca26 x1846780895203968/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234167 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234145.202004:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895203968/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234167 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234145.202012:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234145.202014:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234145.202017:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895203968:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234145.202022:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234142/real 1761234145] req@0000000040d5f361 x1846780895203648/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234167 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234145.202032:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895203648/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234167 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234145.202038:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234145.202040:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234145.202043:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895203648:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761234145.202048:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234142/real 1761234145] req@000000009d315fa5 x1846780895203712/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234167 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234145.202057:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895203712/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234167 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234145.202063:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234145.202066:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234145.202068:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895203712:192.168.201.157@tcp:38:kworker/u8:3.0 00010000:00010000:3.0:1761234145.969344:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234146.993502:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761234147.249323:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234147.249333:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761234147.249337:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234147.249348:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6708 00000100:00080000:2.0:1761234147.249352:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234147.249358:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761234147.249389:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895204096/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234147.249411:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234147.249415:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234147.249416:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234147.249419:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6708 00000100:00080000:2.0:1761234147.249421:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234147.249424:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234147.249433:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234147.249442:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895204160/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234147.249449:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234147.249451:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234147.249452:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234147.249454:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6708 00000100:00080000:2.0:1761234147.249455:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234147.249457:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234147.249461:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234147.249467:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895204224/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234147.249473:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234147.249476:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234147.249477:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234147.249480:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6708 00000100:00080000:2.0:1761234147.249482:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234147.249484:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234147.249487:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234147.249493:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895204288/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234147.249499:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761234147.249505:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895204096:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761234147.249506:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234147.249509:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234147.249518:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6708 00000100:00080000:2.0:1761234147.249520:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234147.249525:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234147.249529:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234147.249543:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895204352/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234147.249567:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234147.249571:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234147.249573:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234147.249576:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6708 00000100:00080000:2.0:1761234147.249579:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234147.249582:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234147.249587:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234147.249598:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895204416/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761234147.249606:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761234147.249612:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234147.249633:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234147.249635:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234147.249640:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6708 00000100:00080000:2.0:1761234147.249642:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234147.249646:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761234147.249649:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234147/real 1761234147] req@00000000fd3935a6 x1846780895204096/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234172 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:2.0:1761234147.249653:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234147.249668:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895204480/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234147.249675:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895204096/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234172 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234147.249690:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234147.249693:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234147.249699:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895204096:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761234147.249707:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895204160:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761234147.249732:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234147.249741:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234147/real 1761234147] req@00000000135e9a6e x1846780895204160/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234172 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234147.249751:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895204160/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234172 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234147.249758:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234147.249761:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234147.249764:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895204160:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234147.249769:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895204224:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234147.249790:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234147.249798:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234147/real 1761234147] req@000000006ff1fdbc x1846780895204224/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234172 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234147.249810:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895204224/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234172 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234147.249819:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234147.249821:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234147.249823:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895204224:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234147.249853:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895204288:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234147.249873:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234147.249881:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234147/real 1761234147] req@0000000088939481 x1846780895204288/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234172 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234147.249890:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895204288/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234172 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234147.249897:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234147.249899:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234147.249902:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895204288:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234147.249907:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895204352:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761234147.249926:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234147.249933:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234147/real 1761234147] req@000000006d8c3eb0 x1846780895204352/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234172 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234147.249942:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895204352/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234172 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234147.249949:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234147.249953:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234147.249955:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895204352:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234147.249961:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895204416:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234147.249980:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234147.249996:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234147/real 1761234147] req@000000006c1041ba x1846780895204416/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234172 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234147.250008:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895204416/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234172 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234147.250015:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234147.250017:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234147.250020:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895204416:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234147.250024:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895204480:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234147.250044:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234147.250051:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234147/real 1761234147] req@00000000fd4aa3a2 x1846780895204480/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234172 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234147.250061:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895204480/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234172 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234147.250068:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234147.250070:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234147.250073:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895204480:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761234148.017384:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234149.041474:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234150.065413:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234151.089432:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234152.113364:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761234152.369332:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234152.369341:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761234152.369345:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234152.369355:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6713 00000100:00080000:3.0:1761234152.369358:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234152.369364:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761234152.369395:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000bbf6af7b x1846780895204544/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234152.369414:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234152.369417:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234152.369418:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234152.369420:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6713 00000100:00080000:3.0:1761234152.369421:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234152.369424:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234152.369431:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234152.369439:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009706a782 x1846780895204608/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761234152.369445:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895204544:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:3.0:1761234152.369448:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234152.369452:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234152.369454:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234152.369458:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6713 00000100:00080000:3.0:1761234152.369460:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234152.369461:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234152.369464:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234152.369470:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007aee79ba x1846780895204672/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234152.369476:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234152.369478:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234152.369479:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234152.369481:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6713 00000100:00080000:3.0:1761234152.369482:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234152.369484:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234152.369486:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234152.369494:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000057e55ffa x1846780895204736/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234152.369499:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234152.369501:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234152.369502:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234152.369503:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6713 00000100:00080000:3.0:1761234152.369505:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234152.369507:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234152.369510:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234152.369515:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002a0ba65f x1846780895204800/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234152.369519:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234152.369521:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234152.369522:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234152.369526:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6713 00000800:00000100:2.0:1761234152.369529:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761234152.369529:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234152.369534:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234152.369537:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234152.369544:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c9993cd4 x1846780895204864/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234152.369549:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234152.369551:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234152.369552:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234152.369554:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6713 00000100:00080000:3.0:1761234152.369555:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1761234152.369558:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234152/real 1761234152] req@00000000bbf6af7b x1846780895204544/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234177 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234152.369561:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234152.369565:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234152.369573:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002625ec0b x1846780895204928/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:2.0:1761234152.369588:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000bbf6af7b x1846780895204544/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234177 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234152.369599:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234152.369603:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234152.369607:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000bbf6af7b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895204544:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:2.0:1761234152.369634:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895204608:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761234152.369659:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234152.369668:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234152/real 1761234152] req@000000009706a782 x1846780895204608/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234177 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234152.369680:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009706a782 x1846780895204608/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234177 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234152.369689:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234152.369691:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234152.369694:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009706a782 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895204608:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761234152.369700:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895204672:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761234152.369721:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234152.369730:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234152/real 1761234152] req@000000007aee79ba x1846780895204672/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234177 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234152.369741:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007aee79ba x1846780895204672/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234177 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234152.369750:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234152.369752:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234152.369755:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007aee79ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895204672:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761234152.369761:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895204736:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761234152.369782:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234152.369790:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234152/real 1761234152] req@0000000057e55ffa x1846780895204736/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234177 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234152.369802:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000057e55ffa x1846780895204736/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234177 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234152.369809:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234152.369812:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234152.369815:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000057e55ffa pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895204736:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761234152.369820:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895204800:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:2.0:1761234152.369840:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234152.369848:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234152/real 1761234152] req@000000002a0ba65f x1846780895204800/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234177 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234152.369859:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002a0ba65f x1846780895204800/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234177 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234152.369874:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234152.369877:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234152.369880:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002a0ba65f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895204800:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:2.0:1761234152.369887:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895204864:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761234152.369908:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234152.369916:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234152/real 1761234152] req@00000000c9993cd4 x1846780895204864/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234177 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234152.369927:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c9993cd4 x1846780895204864/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234177 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234152.369935:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234152.369937:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234152.369940:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c9993cd4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895204864:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:2.0:1761234152.369946:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895204928:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:2.0:1761234152.369966:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234152.369974:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234152/real 1761234152] req@000000002625ec0b x1846780895204928/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234177 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234152.369984:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002625ec0b x1846780895204928/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234177 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:2.0:1761234152.369992:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234152.369994:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:2.0:1761234152.369997:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002625ec0b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895204928:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761234153.137377:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234154.161342:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234155.185427:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234156.209438:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234157.233394:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761234157.489396:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234157.489405:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761234157.489409:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234157.489418:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6718 00000100:00080000:2.0:1761234157.489422:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234157.489427:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761234157.489458:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895204992/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234157.489487:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234157.489490:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234157.489492:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234157.489495:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6718 00000100:00080000:2.0:1761234157.489498:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234157.489501:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234157.489511:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234157.489524:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895205056/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234157.489535:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761234157.489536:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895204992:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:2.0:1761234157.489539:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234157.489542:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234157.489549:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6718 00000100:00080000:2.0:1761234157.489551:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234157.489554:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234157.489559:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234157.489572:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895205120/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234157.489578:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234157.489581:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234157.489582:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234157.489585:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6718 00000100:00080000:2.0:1761234157.489586:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234157.489590:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234157.489594:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234157.489602:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895205184/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234157.489607:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234157.489610:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234157.489611:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234157.489613:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6718 00000100:00080000:2.0:1761234157.489614:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234157.489617:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234157.489619:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761234157.489623:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234157.489629:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895205248/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234157.489634:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234157.489636:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234157.489638:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234157.489641:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6718 00000100:00080000:2.0:1761234157.489642:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234157.489645:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234157.489648:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234157.489653:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234157/real 1761234157] req@000000004ae5bc9b x1846780895204992/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234182 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234157.489655:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895205312/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234157.489673:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234157.489679:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234157.489681:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234157.489684:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6718 00000100:00080000:2.0:1761234157.489686:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234157.489688:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234157.489690:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234157.489692:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895204992/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234182 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:2.0:1761234157.489697:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895205376/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234157.489724:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234157.489729:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234157.489735:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895204992:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761234157.489762:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895205056:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761234157.489790:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234157.489800:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234157/real 1761234157] req@00000000ac7d8d5b x1846780895205056/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234182 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234157.489812:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895205056/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234182 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234157.489821:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234157.489823:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234157.489827:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895205056:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761234157.489833:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895205120:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761234157.489856:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234157.489865:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234157/real 1761234157] req@00000000f4e415d3 x1846780895205120/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234182 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234157.489877:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895205120/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234182 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234157.489885:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234157.489887:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234157.489890:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895205120:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761234157.489897:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895205184:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761234157.489918:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234157.489926:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234157/real 1761234157] req@0000000099b7e5d9 x1846780895205184/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234182 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234157.489938:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895205184/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234182 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234157.489947:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234157.489949:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234157.489952:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895205184:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761234157.489958:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895205248:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:3.0:1761234157.489978:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234157.490005:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234157/real 1761234157] req@00000000c10cebe6 x1846780895205248/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234182 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234157.490019:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895205248/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234182 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234157.490026:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234157.490029:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234157.490031:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895205248:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:3.0:1761234157.490038:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895205312:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761234157.490060:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234157.490068:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234157/real 1761234157] req@00000000a1e8c6bb x1846780895205312/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234182 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234157.490080:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895205312/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234182 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234157.490087:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234157.490089:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234157.490092:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895205312:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:3.0:1761234157.490098:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895205376:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:3.0:1761234157.490118:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234157.490126:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234157/real 1761234157] req@000000008ebccab5 x1846780895205376/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234182 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234157.490136:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895205376/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234182 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234157.490144:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234157.490146:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234157.490148:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895205376:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761234158.257429:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234159.281408:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234160.305427:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234161.329425:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234162.353432:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761234162.609308:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234162.609320:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761234162.609324:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234162.609335:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6723 00000100:00080000:2.0:1761234162.609339:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234162.609346:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761234162.609383:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008ebccab5 x1846780895205440/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234162.609411:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234162.609414:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234162.609416:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234162.609419:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6723 00000100:00080000:2.0:1761234162.609421:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234162.609425:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234162.609436:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234162.609448:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a1e8c6bb x1846780895205504/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234162.609458:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761234162.609461:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895205440:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761234162.609463:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234162.609467:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234162.609473:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6723 00000100:00080000:2.0:1761234162.609476:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234162.609479:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234162.609485:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234162.609496:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000c10cebe6 x1846780895205568/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234162.609505:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234162.609508:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234162.609510:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234162.609513:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6723 00000100:00080000:2.0:1761234162.609515:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234162.609519:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234162.609524:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761234162.609530:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234162.609535:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000099b7e5d9 x1846780895205632/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234162.609544:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234162.609548:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234162.609550:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1761234162.609554:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234162/real 1761234162] req@000000008ebccab5 x1846780895205440/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234187 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234162.609556:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6723 00000100:00080000:2.0:1761234162.609561:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234162.609567:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234162.609572:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234162.609574:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008ebccab5 x1846780895205440/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234187 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234162.609583:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1761234162.609584:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f4e415d3 x1846780895205696/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234162.609586:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234162.609593:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008ebccab5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895205440:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761234162.609603:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234162.609607:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234162.609609:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234162.609614:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6723 00000100:00080000:2.0:1761234162.609616:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761234162.609618:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895205504:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:2.0:1761234162.609623:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234162.609628:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1761234162.609640:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:2.0:1761234162.609643:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000ac7d8d5b x1846780895205760/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234162.609650:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234162/real 1761234162] req@00000000a1e8c6bb x1846780895205504/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234187 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234162.609659:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761234162.609664:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a1e8c6bb x1846780895205504/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234187 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234162.609665:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234162.609667:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234162.609675:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6723 00000100:00080000:3.0:1761234162.609675:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234162.609679:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234162.609679:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00080000:2.0:1761234162.609686:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761234162.609687:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a1e8c6bb pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895205504:192.168.201.157@tcp:38:kworker/u8:3.0 00000008:00100000:2.0:1761234162.609694:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234162.609696:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895205568:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:2.0:1761234162.609710:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004ae5bc9b x1846780895205824/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761234162.609716:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234162.609722:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234162/real 1761234162] req@00000000c10cebe6 x1846780895205568/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234187 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234162.609746:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000c10cebe6 x1846780895205568/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234187 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234162.609751:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234162.609752:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234162.609754:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000c10cebe6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895205568:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234162.609759:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895205632:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234162.609769:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234162.609773:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234162/real 1761234162] req@0000000099b7e5d9 x1846780895205632/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234187 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234162.609779:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000099b7e5d9 x1846780895205632/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234187 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234162.609787:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234162.609788:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234162.609790:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000099b7e5d9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895205632:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234162.609793:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895205696:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761234162.609802:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234162.609806:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234162/real 1761234162] req@00000000f4e415d3 x1846780895205696/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234187 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234162.609822:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f4e415d3 x1846780895205696/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234187 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234162.609827:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234162.609828:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234162.609830:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f4e415d3 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895205696:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234162.609843:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895205760:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234162.609853:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234162.609857:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234162/real 1761234162] req@00000000ac7d8d5b x1846780895205760/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234187 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234162.609864:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000ac7d8d5b x1846780895205760/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234187 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234162.609868:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234162.609869:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234162.609870:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000ac7d8d5b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895205760:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234162.609874:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895205824:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234162.609883:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234162.609887:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234162/real 1761234162] req@000000004ae5bc9b x1846780895205824/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234187 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234162.609907:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004ae5bc9b x1846780895205824/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234187 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234162.609911:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234162.609913:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234162.609914:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004ae5bc9b pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895205824:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761234163.377406:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234164.401402:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234165.425423:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234166.449431:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234167.473347:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761234167.729319:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234167.729330:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761234167.729335:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234167.729347:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6728 00000100:00080000:2.0:1761234167.729351:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234167.729357:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761234167.729392:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895205888/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234167.729421:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234167.729425:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234167.729427:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234167.729432:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6728 00000100:00080000:2.0:1761234167.729434:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234167.729438:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234167.729448:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761234167.729456:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895205888:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761234167.729462:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895205952/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234167.729474:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234167.729477:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234167.729479:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234167.729483:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6728 00000100:00080000:2.0:1761234167.729486:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234167.729490:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234167.729497:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234167.729507:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895206016/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234167.729516:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234167.729519:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234167.729521:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1761234167.729525:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761234167.729529:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6728 00000100:00080000:2.0:1761234167.729532:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234167.729536:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234167.729540:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234167.729551:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895206080/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761234167.729555:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234167/real 1761234167] req@00000000fd4aa3a2 x1846780895205888/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234192 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234167.729578:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1761234167.729584:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895205888/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234192 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234167.729584:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234167.729588:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234167.729597:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6728 00000100:00080000:0.0:1761234167.729598:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234167.729601:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761234167.729602:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761234167.729607:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895205888:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761234167.729610:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234167.729617:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234167.729627:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895206144/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761234167.729632:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895205952:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:2.0:1761234167.729647:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234167.729651:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234167.729653:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234167.729659:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6728 00000800:00000100:0.0:1761234167.729660:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761234167.729663:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234167.729670:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:0.0:1761234167.729671:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234167/real 1761234167] req@000000006c1041ba x1846780895205952/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234192 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000008:00100000:2.0:1761234167.729679:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761234167.729683:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895205952/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234192 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761234167.729690:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761234167.729692:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761234167.729697:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895205952:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:2.0:1761234167.729698:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895206208/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761234167.729704:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895206016:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761234167.729716:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234167.729720:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234167.729722:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234167.729727:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6728 00000100:00080000:2.0:1761234167.729730:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1761234167.729734:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761234167.729735:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234167.729742:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761234167.729749:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234167/real 1761234167] req@000000006d8c3eb0 x1846780895206016/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234192 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234167.729755:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895206272/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:0.0:1761234167.729777:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895206016/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234192 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761234167.729787:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761234167.729790:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761234167.729827:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895206016:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761234167.729834:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895206080:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761234167.729857:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761234167.729866:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234167/real 1761234167] req@0000000088939481 x1846780895206080/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234192 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761234167.729878:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895206080/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234192 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761234167.729886:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761234167.729888:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761234167.729891:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895206080:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761234167.729910:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895206144:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:0.0:1761234167.729931:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761234167.729940:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234167/real 1761234167] req@000000006ff1fdbc x1846780895206144/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234192 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761234167.729951:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895206144/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234192 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761234167.729959:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761234167.729961:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761234167.729965:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895206144:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:0.0:1761234167.729971:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895206208:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761234167.729991:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761234167.729999:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234167/real 1761234167] req@00000000135e9a6e x1846780895206208/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234192 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761234167.730010:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895206208/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234192 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761234167.730018:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761234167.730020:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761234167.730023:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895206208:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:0.0:1761234167.730029:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895206272:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:0.0:1761234167.730048:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:0.0:1761234167.730056:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234167/real 1761234167] req@00000000fd3935a6 x1846780895206272/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234192 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761234167.730067:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895206272/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234192 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761234167.730074:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761234167.730077:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:0.0:1761234167.730080:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895206272:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761234168.497360:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234169.521321:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234170.545340:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234171.569378:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234172.593440:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761234172.849313:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761234172.849322:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761234172.849326:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761234172.849336:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6733 00000100:00080000:0.0:1761234172.849340:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761234172.849345:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761234172.849382:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d1c7f73 x1846780895206336/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761234172.849407:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761234172.849409:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761234172.849410:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761234172.849413:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6733 00000100:00080000:0.0:1761234172.849414:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761234172.849416:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761234172.849423:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761234172.849430:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d67bb589 x1846780895206400/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761234172.849436:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761234172.849438:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761234172.849439:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761234172.849441:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6733 00000100:00080000:0.0:1761234172.849442:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761234172.849444:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761234172.849446:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761234172.849450:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895206464/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761234172.849455:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761234172.849456:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761234172.849457:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761234172.849459:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6733 00000100:00080000:0.0:1761234172.849460:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761234172.849462:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761234172.849465:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761234172.849470:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895206528/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761234172.849476:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761234172.849478:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761234172.849479:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761234172.849481:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6733 00000100:00080000:0.0:1761234172.849483:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761234172.849485:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761234172.849488:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761234172.849494:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895206592/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761234172.849500:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761234172.849503:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895206336:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761234172.849504:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761234172.849507:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761234172.849512:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6733 00000100:00080000:0.0:1761234172.849513:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761234172.849515:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761234172.849518:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761234172.849526:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895206656/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761234172.849531:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761234172.849533:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761234172.849535:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761234172.849538:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6733 00000100:00080000:0.0:1761234172.849539:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761234172.849541:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761234172.849543:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761234172.849549:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895206720/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761234172.849569:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234172.849589:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234172/real 1761234172] req@000000009d1c7f73 x1846780895206336/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234197 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234172.849600:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d1c7f73 x1846780895206336/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234197 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234172.849607:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234172.849609:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234172.849612:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d1c7f73 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895206336:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761234172.849616:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895206400:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761234172.849629:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234172.849633:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234172/real 1761234172] req@00000000d67bb589 x1846780895206400/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234197 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234172.849640:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d67bb589 x1846780895206400/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234197 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234172.849644:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234172.849645:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234172.849647:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d67bb589 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895206400:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234172.849650:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895206464:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234172.849661:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234172.849664:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234172/real 1761234172] req@00000000a4a2c44a x1846780895206464/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234197 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234172.849670:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895206464/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234197 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234172.849675:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234172.849676:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234172.849677:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895206464:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234172.849680:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895206528:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234172.849689:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234172.849692:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234172/real 1761234172] req@000000004a74da19 x1846780895206528/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234197 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234172.849729:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895206528/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234197 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234172.849738:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234172.849740:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234172.849741:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895206528:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234172.849764:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895206592:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761234172.849778:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234172.849783:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234172/real 1761234172] req@000000002d1b8ef7 x1846780895206592/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234197 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234172.849789:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895206592/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234197 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234172.849793:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234172.849795:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234172.849796:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895206592:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234172.849799:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895206656:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234172.849808:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234172.849812:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234172/real 1761234172] req@000000007bd1ec64 x1846780895206656/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234197 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234172.849818:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895206656/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234197 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234172.849822:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234172.849823:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234172.849824:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895206656:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234172.849828:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895206720:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234172.849837:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234172.849840:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234172/real 1761234172] req@000000001db96463 x1846780895206720/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234197 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234172.849847:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895206720/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234197 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234172.849853:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234172.849854:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234172.849857:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895206720:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761234173.617438:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234174.641478:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234175.665406:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234176.689452:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234177.713524:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761234177.969414:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234177.969427:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761234177.969432:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234177.969445:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6738 00000100:00080000:2.0:1761234177.969449:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234177.969458:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761234177.969499:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895206784/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234177.969530:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234177.969535:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234177.969536:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234177.969540:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6738 00000100:00080000:2.0:1761234177.969542:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234177.969546:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234177.969555:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234177.969568:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895206848/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234177.969580:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761234177.969586:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895206784:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761234177.969587:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234177.969590:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234177.969619:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6738 00000100:00080000:2.0:1761234177.969622:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234177.969626:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234177.969632:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234177.969645:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895206912/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234177.969655:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234177.969659:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234177.969661:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234177.969664:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6738 00000100:00080000:2.0:1761234177.969667:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234177.969671:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234177.969674:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234177.969689:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895206976/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761234177.969700:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:2.0:1761234177.969708:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234177.969713:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234177.969715:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234177.969722:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6738 00000100:00080000:2.0:1761234177.969724:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234177.969728:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234177.969733:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234177.969736:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234177/real 1761234177] req@000000009d315fa5 x1846780895206784/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234202 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234177.969748:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895207040/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234177.969762:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1761234177.969767:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895206784/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234202 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:2.0:1761234177.969769:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234177.969773:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234177.969784:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6738 00000100:00080000:2.0:1761234177.969787:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234177.969789:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1761234177.969792:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:3.0:1761234177.969795:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000008:00100000:2.0:1761234177.969799:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234177.969805:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895206784:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:2.0:1761234177.969813:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895207104/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234177.969824:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234177.969827:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234177.969829:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234177.969834:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6738 00000100:00080000:2.0:1761234177.969837:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761234177.969841:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895206848:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:2.0:1761234177.969847:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234177.969856:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234177.969872:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895207168/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000800:00000100:3.0:1761234177.969883:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234177.969904:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234177/real 1761234177] req@0000000040d5f361 x1846780895206848/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234202 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234177.969923:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895206848/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234202 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234177.969930:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234177.969933:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234177.969936:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895206848:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234177.969942:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895206912:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234177.969960:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234177.969966:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234177/real 1761234177] req@000000004f29ca26 x1846780895206912/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234202 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234177.969976:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895206912/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234202 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234177.969983:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234177.969985:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234177.969988:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895206912:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234177.969993:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895206976:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234177.970008:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234177.970014:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234177/real 1761234177] req@000000007091f547 x1846780895206976/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234202 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234177.970024:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895206976/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234202 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234177.970032:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234177.970035:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234177.970037:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895206976:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234177.970043:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895207040:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761234177.970058:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234177.970064:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234177/real 1761234177] req@0000000064a2cea0 x1846780895207040/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234202 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234177.970074:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895207040/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234202 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234177.970081:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234177.970083:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234177.970085:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895207040:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234177.970090:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895207104:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234177.970106:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234177.970112:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234177/real 1761234177] req@0000000033765045 x1846780895207104/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234202 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234177.970121:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895207104/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234202 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234177.970128:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234177.970130:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234177.970133:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895207104:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234177.970156:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895207168:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234177.970176:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234177.970182:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234177/real 1761234177] req@000000007dcf84ae x1846780895207168/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234202 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234177.970192:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895207168/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234202 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234177.970199:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234177.970233:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234177.970236:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895207168:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761234178.737469:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234179.761431:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234180.785398:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234181.809425:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234182.833383:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1761234183.089330:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761234183.089339:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:0.0:1761234183.089342:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761234183.089352:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6744 00000100:00080000:0.0:1761234183.089355:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761234183.089360:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:0.0:1761234183.089391:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000005d83493f x1846780895207232/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761234183.089412:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761234183.089414:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761234183.089416:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761234183.089418:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6744 00000100:00080000:0.0:1761234183.089419:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761234183.089422:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761234183.089429:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761234183.089437:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000f3ed2588 x1846780895207296/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761234183.089444:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761234183.089447:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761234183.089448:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761234183.089452:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6744 00000100:00100000:3.0:1761234183.089453:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895207232:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:0.0:1761234183.089454:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761234183.089458:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761234183.089464:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761234183.089473:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001db96463 x1846780895207360/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761234183.089478:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761234183.089480:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761234183.089481:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761234183.089485:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6744 00000100:00080000:0.0:1761234183.089486:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761234183.089488:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761234183.089491:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761234183.089533:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007bd1ec64 x1846780895207424/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761234183.089543:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1761234183.089545:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:0.0:1761234183.089547:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1761234183.089551:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761234183.089556:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6744 00000100:00080000:0.0:1761234183.089559:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761234183.089564:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761234183.089569:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234183.089573:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234183/real 1761234183] req@000000005d83493f x1846780895207232/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234208 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:0.0:1761234183.089585:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000002d1b8ef7 x1846780895207488/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234183.089593:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000005d83493f x1846780895207232/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234208 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:0.0:1761234183.089598:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761234183.089605:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1761234183.089607:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761234183.089613:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6744 00000100:00080000:0.0:1761234183.089614:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234183.089616:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1761234183.089618:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00080000:3.0:1761234183.089622:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000008:00100000:0.0:1761234183.089624:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234183.089630:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000005d83493f pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895207232:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:0.0:1761234183.089635:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004a74da19 x1846780895207552/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:0.0:1761234183.089642:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1761234183.089644:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1761234183.089645:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1761234183.089649:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6744 00000100:00080000:0.0:1761234183.089650:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1761234183.089652:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:0.0:1761234183.089654:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1761234183.089662:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a4a2c44a x1846780895207616/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234183.089662:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895207296:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761234183.089697:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234183.089706:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234183/real 1761234183] req@00000000f3ed2588 x1846780895207296/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234208 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234183.089724:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000f3ed2588 x1846780895207296/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234208 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234183.089731:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234183.089734:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234183.089737:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000f3ed2588 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895207296:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234183.089743:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895207360:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234183.089761:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234183.089768:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234183/real 1761234183] req@000000001db96463 x1846780895207360/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234208 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234183.089778:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001db96463 x1846780895207360/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234208 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234183.089784:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234183.089787:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234183.089790:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001db96463 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895207360:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234183.089795:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895207424:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234183.089812:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234183.089819:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234183/real 1761234183] req@000000007bd1ec64 x1846780895207424/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234208 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234183.089829:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007bd1ec64 x1846780895207424/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234208 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234183.089836:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234183.089838:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234183.089841:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007bd1ec64 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895207424:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234183.089847:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895207488:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761234183.089866:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234183.089873:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234183/real 1761234183] req@000000002d1b8ef7 x1846780895207488/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234208 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234183.089883:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000002d1b8ef7 x1846780895207488/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234208 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234183.089889:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234183.089892:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234183.089894:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000002d1b8ef7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895207488:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234183.089899:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895207552:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234183.089914:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234183.089921:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234183/real 1761234183] req@000000004a74da19 x1846780895207552/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234208 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234183.089930:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004a74da19 x1846780895207552/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234208 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234183.089937:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234183.089940:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234183.089943:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004a74da19 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895207552:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234183.089963:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895207616:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234183.089985:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234183.089992:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234183/real 1761234183] req@00000000a4a2c44a x1846780895207616/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234208 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234183.090002:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a4a2c44a x1846780895207616/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234208 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234183.090009:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234183.090012:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234183.090015:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a4a2c44a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895207616:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761234183.857405:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234184.881408:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234185.905463:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234186.929366:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234187.953389:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1761234188.209322:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234188.209332:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:1.0:1761234188.209336:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234188.209346:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6749 00000100:00080000:1.0:1761234188.209350:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234188.209357:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:1.0:1761234188.209390:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000cbad49de x1846780895207680/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234188.209432:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234188.209435:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761234188.209437:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234188.209439:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6749 00000100:00080000:1.0:1761234188.209441:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234188.209443:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234188.209451:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234188.209460:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e467e107 x1846780895207744/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234188.209467:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234188.209469:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1761234188.209470:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234188.209474:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6749 00000100:00080000:1.0:1761234188.209476:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761234188.209476:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000cbad49de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895207680:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:1.0:1761234188.209480:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234188.209486:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234188.209493:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090800dde x1846780895207808/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234188.209498:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234188.209500:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761234188.209501:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234188.209503:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6749 00000100:00080000:1.0:1761234188.209505:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234188.209507:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234188.209509:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234188.209517:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000328368c8 x1846780895207872/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234188.209521:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234188.209523:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1761234188.209524:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234188.209526:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6749 00000100:00080000:1.0:1761234188.209528:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234188.209531:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234188.209533:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234188.209544:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000a611ee34 x1846780895207936/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234188.209549:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234188.209555:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1761234188.209555:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:1.0:1761234188.209557:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234188.209562:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6749 00000100:00080000:1.0:1761234188.209570:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1761234188.209572:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234188.209576:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234188.209582:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234188/real 1761234188] req@00000000cbad49de x1846780895207680/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234213 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234188.209587:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000001aedf9b1 x1846780895208000/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234188.209611:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234188.209614:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1761234188.209615:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1761234188.209618:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6749 00000100:00080000:1.0:1761234188.209619:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1761234188.209623:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000cbad49de x1846780895207680/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234213 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234188.209624:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:1.0:1761234188.209629:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234188.209638:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000d74b2e3c x1846780895208064/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234188.209641:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234188.209646:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234188.209653:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000cbad49de pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895207680:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761234188.209680:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e467e107 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895207744:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761234188.209705:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234188.209715:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234188/real 1761234188] req@00000000e467e107 x1846780895207744/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234213 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234188.209735:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e467e107 x1846780895207744/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234213 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234188.209744:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234188.209746:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234188.209749:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e467e107 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895207744:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234188.209755:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090800dde pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895207808:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234188.209776:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234188.209784:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234188/real 1761234188] req@0000000090800dde x1846780895207808/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234213 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234188.209795:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090800dde x1846780895207808/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234213 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234188.209803:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234188.209805:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234188.209808:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090800dde pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895207808:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234188.209814:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895207872:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234188.209834:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234188.209843:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234188/real 1761234188] req@00000000328368c8 x1846780895207872/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234213 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234188.209854:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000328368c8 x1846780895207872/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234213 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234188.209861:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234188.209864:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234188.209867:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000328368c8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895207872:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234188.209873:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895207936:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761234188.209893:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234188.209901:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234188/real 1761234188] req@00000000a611ee34 x1846780895207936/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234213 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234188.209911:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000a611ee34 x1846780895207936/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234213 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234188.209918:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234188.209920:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234188.209923:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000a611ee34 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895207936:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234188.209929:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895208000:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234188.209948:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234188.209956:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234188/real 1761234188] req@000000001aedf9b1 x1846780895208000/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234213 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234188.209967:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000001aedf9b1 x1846780895208000/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234213 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234188.209974:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234188.209977:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234188.209980:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000001aedf9b1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895208000:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234188.209985:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895208064:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234188.210005:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234188.210012:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234188/real 1761234188] req@00000000d74b2e3c x1846780895208064/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234213 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234188.210021:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000d74b2e3c x1846780895208064/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234213 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234188.210028:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234188.210030:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234188.210032:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000d74b2e3c pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895208064:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761234188.977429:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234190.001414:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234191.025383:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234192.049438:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234193.073380:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761234193.329360:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234193.329368:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761234193.329372:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234193.329381:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6754 00000100:00080000:2.0:1761234193.329384:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234193.329390:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761234193.329422:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007dcf84ae x1846780895208128/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234193.329443:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234193.329445:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234193.329447:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234193.329449:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6754 00000100:00080000:2.0:1761234193.329451:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234193.329453:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234193.329459:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234193.329470:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000033765045 x1846780895208192/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234193.329476:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234193.329478:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234193.329480:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234193.329482:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6754 00000100:00080000:2.0:1761234193.329483:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234193.329489:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:3.0:1761234193.329489:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895208128:192.168.201.157@tcp:250:kworker/u8:3.0 00000008:00100000:2.0:1761234193.329493:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234193.329502:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000064a2cea0 x1846780895208256/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234193.329508:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234193.329509:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234193.329510:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234193.329512:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6754 00000100:00080000:2.0:1761234193.329513:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234193.329516:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234193.329519:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234193.329525:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007091f547 x1846780895208320/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234193.329530:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234193.329533:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234193.329534:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234193.329536:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6754 00000100:00080000:2.0:1761234193.329537:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234193.329540:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234193.329542:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234193.329547:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000004f29ca26 x1846780895208384/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234193.329553:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234193.329555:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234193.329557:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234193.329558:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6754 00000100:00080000:2.0:1761234193.329559:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234193.329564:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000800:00000100:3.0:1761234193.329566:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000008:00100000:2.0:1761234193.329568:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234193.329576:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000040d5f361 x1846780895208448/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234193.329581:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234193.329583:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234193.329584:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234193.329586:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6754 00000100:00080000:2.0:1761234193.329588:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234193.329590:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234193.329592:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234193.329594:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234193/real 1761234193] req@000000007dcf84ae x1846780895208128/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234218 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:2.0:1761234193.329601:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000009d315fa5 x1846780895208512/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:3.0:1761234193.329629:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007dcf84ae x1846780895208128/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234218 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234193.329641:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234193.329644:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234193.329648:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007dcf84ae pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895208128:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:3.0:1761234193.329656:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895208192:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761234193.329677:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234193.329685:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234193/real 1761234193] req@0000000033765045 x1846780895208192/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234218 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234193.329701:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000033765045 x1846780895208192/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234218 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234193.329709:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234193.329711:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234193.329714:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000033765045 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895208192:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234193.329740:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895208256:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234193.329759:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234193.329766:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234193/real 1761234193] req@0000000064a2cea0 x1846780895208256/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234218 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234193.329776:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000064a2cea0 x1846780895208256/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234218 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234193.329784:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234193.329786:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234193.329789:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000064a2cea0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895208256:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234193.329794:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895208320:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234193.329810:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234193.329817:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234193/real 1761234193] req@000000007091f547 x1846780895208320/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234218 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234193.329826:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007091f547 x1846780895208320/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234218 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234193.329833:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234193.329836:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234193.329838:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007091f547 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895208320:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234193.329843:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895208384:192.168.201.157@tcp:38:kworker/u8:3.0 00000800:00000100:3.0:1761234193.329860:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234193.329867:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234193/real 1761234193] req@000000004f29ca26 x1846780895208384/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234218 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234193.329877:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000004f29ca26 x1846780895208384/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234218 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234193.329884:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234193.329886:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234193.329889:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000004f29ca26 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895208384:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:3.0:1761234193.329894:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895208448:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234193.329910:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234193.329917:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234193/real 1761234193] req@0000000040d5f361 x1846780895208448/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234218 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234193.329926:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000040d5f361 x1846780895208448/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234218 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234193.329934:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234193.329936:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234193.329939:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000040d5f361 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895208448:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:3.0:1761234193.329944:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895208512:192.168.201.157@tcp:8:kworker/u8:3.0 00000800:00000100:3.0:1761234193.329962:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234193.329968:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234193/real 1761234193] req@000000009d315fa5 x1846780895208512/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234218 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:3.0:1761234193.329978:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000009d315fa5 x1846780895208512/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234218 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:3.0:1761234193.329984:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234193.329986:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:3.0:1761234193.329989:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000009d315fa5 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895208512:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:1.0:1761234194.097410:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234195.121471:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234196.145368:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234197.169459:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234198.193427:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761234198.449335:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234198.449346:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761234198.449350:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234198.449363:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6759 00000100:00080000:3.0:1761234198.449367:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234198.449377:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761234198.449416:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000003b40830d x1846780895208576/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234198.449447:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234198.449451:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234198.449453:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234198.449457:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6759 00000100:00080000:3.0:1761234198.449459:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234198.449465:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234198.449476:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234198.449491:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000829a5595 x1846780895208640/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761234198.449500:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895208576:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:3.0:1761234198.449508:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234198.449513:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234198.449514:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234198.449519:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6759 00000100:00080000:3.0:1761234198.449522:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234198.449525:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234198.449531:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234198.449546:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000089b8c4f1 x1846780895208704/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234198.449558:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234198.449561:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234198.449563:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234198.449568:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6759 00000100:00080000:3.0:1761234198.449570:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234198.449575:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234198.449580:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234198.449590:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000070fa3bd6 x1846780895208768/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234198.449598:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234198.449601:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234198.449602:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1761234198.449608:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00080000:3.0:1761234198.449609:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6759 00000100:00080000:3.0:1761234198.449614:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234198.449619:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234198.449624:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234198.449636:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090702bc4 x1846780895208832/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234198.449645:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1761234198.449651:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234198/real 1761234198] req@000000003b40830d x1846780895208576/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234223 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:3.0:1761234198.449653:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234198.449656:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234198.449668:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6759 00000100:00080000:3.0:1761234198.449673:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234198.449681:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234198.449687:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234198.449692:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000003b40830d x1846780895208576/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234223 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:3.0:1761234198.449701:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6fdd7f9 x1846780895208896/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:1.0:1761234198.449709:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1761234198.449715:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1761234198.449718:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00080000:3.0:1761234198.449721:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234198.449725:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761234198.449730:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000003b40830d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895208576:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00080000:3.0:1761234198.449731:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6759 00000100:00080000:3.0:1761234198.449738:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234198.449743:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234198.449748:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234198.449760:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008e58c4b8 x1846780895208960/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761234198.449779:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895208640:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761234198.449811:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234198.449820:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234198/real 1761234198] req@00000000829a5595 x1846780895208640/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234223 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234198.449832:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000829a5595 x1846780895208640/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234223 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234198.449839:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234198.449842:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234198.449845:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000829a5595 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895208640:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761234198.449851:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895208704:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234198.449871:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234198.449878:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234198/real 1761234198] req@0000000089b8c4f1 x1846780895208704/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234223 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234198.449888:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000089b8c4f1 x1846780895208704/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234223 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234198.449894:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234198.449896:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234198.449899:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000089b8c4f1 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895208704:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234198.449905:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895208768:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234198.449923:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234198.449931:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234198/real 1761234198] req@0000000070fa3bd6 x1846780895208768/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234223 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234198.449941:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895208768/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234223 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234198.449947:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234198.449949:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234198.449952:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895208768:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234198.449958:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895208832:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761234198.449977:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234198.449985:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234198/real 1761234198] req@0000000090702bc4 x1846780895208832/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234223 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234198.449994:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895208832/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234223 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234198.450002:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234198.450004:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234198.450007:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895208832:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761234198.450013:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895208896:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234198.450031:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234198.450038:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234198/real 1761234198] req@00000000b6fdd7f9 x1846780895208896/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234223 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234198.450049:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895208896/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234223 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234198.450056:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234198.450059:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234198.450062:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895208896:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234198.450084:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895208960:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234198.450102:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234198.450110:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234198/real 1761234198] req@000000008e58c4b8 x1846780895208960/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234223 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234198.450121:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895208960/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234223 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234198.450130:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234198.450132:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234198.450135:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895208960:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:3.0:1761234199.217564:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234200.241370:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234201.265414:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234202.289338:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234203.313446:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1761234203.569321:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234203.569331:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:3.0:1761234203.569334:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234203.569344:0:47728:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6764 00000100:00080000:3.0:1761234203.569347:0:47728:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234203.569371:0:47728:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:3.0:1761234203.569413:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000e99e5a12 x1846780895209024/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234203.569442:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234203.569446:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234203.569447:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234203.569451:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6764 00000100:00080000:3.0:1761234203.569453:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234203.569456:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234203.569465:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234203.569476:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000007adf99ab x1846780895209088/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234203.569485:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234203.569488:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234203.569489:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234203.569509:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6764 00000100:00080000:3.0:1761234203.569511:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234203.569513:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234203.569517:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1761234203.569520:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895209024:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:3.0:1761234203.569526:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000080131ea7 x1846780895209152/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234203.569533:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234203.569535:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234203.569535:0:47728:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234203.569537:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6764 00000100:00080000:3.0:1761234203.569538:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234203.569540:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234203.569542:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234203.569551:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000008e58c4b8 x1846780895209216/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234203.569556:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234203.569558:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1761234203.569559:0:47728:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234203.569561:0:47728:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6764 00000100:00080000:3.0:1761234203.569562:0:47728:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234203.569564:0:47728:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234203.569567:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234203.569573:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000b6fdd7f9 x1846780895209280/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234203.569578:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234203.569581:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1761234203.569582:0:47728:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234203.569584:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6764 00000100:00080000:3.0:1761234203.569586:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234203.569588:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234203.569590:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1761234203.569597:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:3.0:1761234203.569598:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000090702bc4 x1846780895209344/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:3.0:1761234203.569604:0:47728:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1761234203.569605:0:47728:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1761234203.569607:0:47728:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1761234203.569609:0:47728:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6764 00000100:00080000:3.0:1761234203.569610:0:47728:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1761234203.569613:0:47728:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:3.0:1761234203.569615:0:47728:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1761234203.569621:0:47728:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000070fa3bd6 x1846780895209408/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761234203.569627:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234203/real 1761234203] req@00000000e99e5a12 x1846780895209024/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234228 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234203.569649:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000e99e5a12 x1846780895209024/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234228 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234203.569659:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234203.569662:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234203.569666:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000e99e5a12 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895209024:192.168.201.157@tcp:250:kworker/u8:2.0 00000100:00100000:1.0:1761234203.569673:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895209088:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761234203.569697:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234203.569705:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234203/real 1761234203] req@000000007adf99ab x1846780895209088/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234228 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234203.569716:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000007adf99ab x1846780895209088/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234228 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234203.569723:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234203.569726:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234203.569729:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000007adf99ab pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895209088:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761234203.569755:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895209152:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234203.569776:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234203.569784:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234203/real 1761234203] req@0000000080131ea7 x1846780895209152/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234228 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234203.569794:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000080131ea7 x1846780895209152/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234228 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234203.569801:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234203.569803:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234203.569805:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000080131ea7 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895209152:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234203.569811:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895209216:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234203.569830:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234203.569837:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234203/real 1761234203] req@000000008e58c4b8 x1846780895209216/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234228 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234203.569864:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000008e58c4b8 x1846780895209216/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234228 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234203.569874:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234203.569877:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234203.569881:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000008e58c4b8 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895209216:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234203.569889:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895209280:192.168.201.157@tcp:38:kworker/u8:2.0 00000800:00000100:1.0:1761234203.569915:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234203.569925:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234203/real 1761234203] req@00000000b6fdd7f9 x1846780895209280/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234228 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234203.569941:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000b6fdd7f9 x1846780895209280/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234228 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234203.569951:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234203.569964:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234203.569969:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000b6fdd7f9 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895209280:192.168.201.157@tcp:38:kworker/u8:2.0 00000100:00100000:1.0:1761234203.569977:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895209344:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234203.570005:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234203.570016:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234203/real 1761234203] req@0000000090702bc4 x1846780895209344/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234228 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234203.570055:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000090702bc4 x1846780895209344/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234228 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234203.570066:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234203.570069:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234203.570073:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000090702bc4 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895209344:192.168.201.157@tcp:8:kworker/u8:2.0 00000100:00100000:1.0:1761234203.570082:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895209408:192.168.201.157@tcp:8:kworker/u8:2.0 00000800:00000100:1.0:1761234203.570171:0:2219:0:(socklnd_cb.c:970:ksocknal_launch_packet()) No usable routes to 12345-192.168.201.157@tcp 00000100:00100000:1.0:1761234203.570185:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234203/real 1761234203] req@0000000070fa3bd6 x1846780895209408/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234228 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00100000:1.0:1761234203.570244:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000070fa3bd6 x1846780895209408/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234228 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:2.0' 00000100:00080000:1.0:1761234203.570255:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234203.570258:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234203.570262:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000070fa3bd6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895209408:192.168.201.157@tcp:8:kworker/u8:2.0 00010000:00010000:1.0:1761234204.337378:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234205.361560:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234206.385447:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234207.409468:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234208.433395:0:26364:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1761234208.689351:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 1cb6fe23-cc45-46e1-92eb-f576e5099f9b->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234208.689363:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) MGS: starting recovery 00000100:00080000:2.0:1761234208.689368:0:38:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234208.689381:0:38:0:(import.c:534:import_select_connection()) MGC192.168.201.157@tcp: connect to NID 192.168.201.157@tcp last attempt 6769 00000100:00080000:2.0:1761234208.689384:0:38:0:(import.c:581:import_select_connection()) MGC192.168.201.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234208.689393:0:38:0:(import.c:615:import_select_connection()) MGC192.168.201.157@tcp: import 00000000436f2c03 using connection MGC192.168.201.157@tcp_0/192.168.201.157@tcp 00000100:00100000:2.0:1761234208.689440:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd3935a6 x1846780895209472/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234208.689473:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234208.689476:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234208.689477:0:38:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234208.689480:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6769 00000100:00080000:2.0:1761234208.689482:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234208.689486:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3e004b800: import 00000000917f3867 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234208.689496:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234208.689521:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000135e9a6e x1846780895209536/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761234208.689524:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895209472:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00080000:2.0:1761234208.689543:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234208.689548:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234208.689550:0:38:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234208.689557:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6769 00000100:00080000:2.0:1761234208.689559:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234208.689564:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3e004b800: import 00000000fdb09a18 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234208.689570:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234208.689583:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006ff1fdbc x1846780895209600/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234208.689590:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 20927715-ad47-4343-a831-9a256a8bfb1b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234208.689593:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234208.689594:0:38:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234208.689598:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: connect to NID 192.168.201.157@tcp last attempt 6769 00000100:00080000:2.0:1761234208.689600:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234208.689604:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3e004b800: import 000000003c13195a using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234208.689608:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234208.689615:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@0000000088939481 x1846780895209664/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00080000:2.0:1761234208.689620:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234208.689624:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1761234208.689627:0:38:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761234208.689629:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895209536:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:2.0:1761234208.689641:0:38:0:(import.c:534:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6769 00000100:00080000:2.0:1761234208.689643:0:38:0:(import.c:581:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234208.689646:0:38:0:(import.c:615:import_select_connection()) lustre-MDT0000-mdc-ffff92b3d0860800: import 000000000ce77d7d using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234208.689650:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xae7e5e7be374d3b8 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234208.689671:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006d8c3eb0 x1846780895209728/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761234208.689677:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895209600:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761234208.689698:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234208.689701:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1761234208.689704:0:38:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1761234208.689713:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895209664:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00080000:2.0:1761234208.689717:0:38:0:(import.c:534:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6769 00000100:00080000:2.0:1761234208.689725:0:38:0:(import.c:581:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234208.689733:0:38:0:(import.c:615:import_select_connection()) lustre-OST0000-osc-ffff92b3d0860800: import 0000000082901bab using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000008:00100000:2.0:1761234208.689740:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0465af2e3640078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234208.689755:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@000000006c1041ba x1846780895209792/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761234208.689760:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895209728:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00080000:2.0:1761234208.689788:0:38:0:(pinger.c:251:ptlrpc_pinger_process_import()) 5ac69958-7a77-4161-b126-1e70633ca6ad->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1761234208.689793:0:38:0:(recover.c:57:ptlrpc_initiate_recovery()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1761234208.689796:0:38:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1761234208.689805:0:38:0:(import.c:534:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: connect to NID 192.168.201.157@tcp last attempt 6769 00000100:00080000:2.0:1761234208.689807:0:38:0:(import.c:581:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1761234208.689811:0:38:0:(import.c:615:import_select_connection()) lustre-OST0001-osc-ffff92b3d0860800: import 00000000ee1fdd02 using connection 192.168.201.157@tcp/192.168.201.157@tcp 00000100:00100000:1.0:1761234208.689815:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895209792:192.168.201.157@tcp:8:kworker/u8:3.0 00000008:00100000:2.0:1761234208.689820:0:38:0:(osc_request.c:3485:osc_reconnect()) ocd_connect_flags: 0xa0425af2e3440078 ocd_version: 34539264 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1761234208.689842:0:38:0:(import.c:807:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@00000000fd4aa3a2 x1846780895209856/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' 00000100:00100000:1.0:1761234208.689867:0:2219:0:(client.c:1733:ptlrpc_send_new_req()) Sending RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895209856:192.168.201.157@tcp:8:kworker/u8:3.0 00010000:00010000:3.0:1761234209.457592:0:5001:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1761234210.481372:0:5002:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1761234211.505449:0:16788:0:(ldlm_lockd.c:2792:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1761234211.761543:0:2209:0:(lib-socket.c:430:lnet_sock_connect()) Error -113 connecting 1020 -> 192.168.201.157:988 00000400:00000100:2.0:1761234211.761622:0:2209:0:(acceptor.c:105:lnet_connect_console_error()) Connection to 192.168.201.157@tcp at host 192.168.201.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1761234211.761638:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761234211.761690:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761234211.761702:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761234211.761712:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761234211.761724:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:1.0:1761234211.761733:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234208/real 1761234211] req@00000000135e9a6e x1846780895209536/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234233 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000800:00000100:2.0:1761234211.761735:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000800:00000100:2.0:1761234211.761748:0:2209:0:(socklnd_cb.c:430:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.201.157@tcp 00000100:00100000:1.0:1761234211.761763:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000135e9a6e x1846780895209536/t0(0) o38->lustre-MDT0000-mdc-ffff92b3e004b800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234233 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234211.761775:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000917f3867 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234211.761779:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234211.761784:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000135e9a6e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895209536:192.168.201.157@tcp:38:kworker/u8:3.0 00000100:00100000:1.0:1761234211.761791:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234208/real 1761234211] req@000000006ff1fdbc x1846780895209600/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234233 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234211.761805:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006ff1fdbc x1846780895209600/t0(0) o8->lustre-OST0000-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234233 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234211.761812:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000fdb09a18 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234211.761814:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234211.761817:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006ff1fdbc pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895209600:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761234211.761833:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234208/real 1761234211] req@0000000088939481 x1846780895209664/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234233 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234211.761843:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@0000000088939481 x1846780895209664/t0(0) o8->lustre-OST0001-osc-ffff92b3e004b800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234233 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234211.761851:0:2219:0:(import.c:86:import_set_state_nolock()) 000000003c13195a lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234211.761854:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234211.761857:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@0000000088939481 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:20927715-ad47-4343-a831-9a256a8bfb1b:2219:1846780895209664:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761234211.761863:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234208/real 1761234211] req@00000000fd3935a6 x1846780895209472/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234233 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234211.761875:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd3935a6 x1846780895209472/t0(0) o250->MGC192.168.201.157@tcp@192.168.201.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1761234233 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234211.761883:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000436f2c03 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234211.761885:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of MGS on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234211.761888:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd3935a6 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:1cb6fe23-cc45-46e1-92eb-f576e5099f9b:2219:1846780895209472:192.168.201.157@tcp:250:kworker/u8:3.0 00000100:00100000:1.0:1761234211.761921:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234208/real 1761234211] req@00000000fd4aa3a2 x1846780895209856/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234233 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234211.761933:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@00000000fd4aa3a2 x1846780895209856/t0(0) o8->lustre-OST0001-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234233 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234211.761940:0:2219:0:(import.c:86:import_set_state_nolock()) 00000000ee1fdd02 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234211.761943:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234211.761945:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@00000000fd4aa3a2 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895209856:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761234211.761951:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234208/real 1761234211] req@000000006c1041ba x1846780895209792/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234233 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234211.762006:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006c1041ba x1846780895209792/t0(0) o8->lustre-OST0000-osc-ffff92b3d0860800@192.168.201.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1761234233 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234211.762020:0:2219:0:(import.c:86:import_set_state_nolock()) 0000000082901bab lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234211.762022:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234211.762025:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006c1041ba pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895209792:192.168.201.157@tcp:8:kworker/u8:3.0 00000100:00100000:1.0:1761234211.762030:0:2219:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1761234208/real 1761234211] req@000000006d8c3eb0 x1846780895209728/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234233 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00100000:1.0:1761234211.762040:0:2219:0:(client.c:2326:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@000000006d8c3eb0 x1846780895209728/t0(0) o38->lustre-MDT0000-mdc-ffff92b3d0860800@192.168.201.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1761234233 ref 1 fl Rpc:eXNQU/0/ffffffff rc 0/-1 job:'kworker/u8:3.0' 00000100:00080000:1.0:1761234211.762047:0:2219:0:(import.c:86:import_set_state_nolock()) 000000000ce77d7d lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1761234211.762051:0:2219:0:(import.c:1429:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.201.157@tcp failed (-110) 00000100:00100000:1.0:1761234211.762054:0:2219:0:(client.c:2215:ptlrpc_check_set()) Completed RPC req@000000006d8c3eb0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:5ac69958-7a77-4161-b126-1e70633ca6ad:2219:1846780895209728:192.168.201.157@tcp:38:kworker/u8:3.0 Debug log: 31357 lines, 31357 kept, 0 dropped, 0 bad.